Thursday, September 29, 2016

Be more intelligent with your `sleep`--or-- how to overengineer your scripts

The universal solution to waiting for something to be ready in shell scripting is the `sleep` command.

Here, we're waiting for a dir to be created (say from a `yum install httpd` going on in another terminal), so we can `ls` the contents, perhaps as part of a script that configures httpd.

sleep 6
ls /etc/init.d/httpd

But is there a better way? What if the directory exists almost immediately? You've wasted nearly seconds unnecessarily, which if you do this a lot in your script, adds a bunch of time.

We need a way to continuously query if the resource exists. And it needs to be with a command that sets an error exit code if the resource is not found. Let's use `stat` to do this.

Leveraging the fact that `stat` will set a non-zero exit code on failure

while true; do
  stat /etc/init.d/httpd
  if [ $? -eq 0 ]; then # check if the stat was successful
    break
  fi
done
ls /etc/init.d/httpd

'true' is a command, whose output is nothing and whose exit status is 0. In terms of performance we'd probably like not to call a binary, but `true` is a shell built-in. The no-op operator `:` accomplishes the same thing, ex: `while :; do`.

We can simplify the if statement by using the && operator which executes the following command break, if stat exits without error (sets a 0 status)

while true; do
  stat /etc/init.d/httpd && break
done
ls /etc/init.d/httpd

Instead of stat, we can use the `test` command (aliased as `[`). Here we check for a file using -f.

while true; do
  if [ -f /etc/init.d/httpd ]; then break; fi
done
ls /etc/init.d/httpd

However, in these examples, if the file never exists, the loop will never exit.

So, instead we can define a timeout and a shorter `sleep` interval, and a counter (i) to track the iterations:

i=0
while [ "$i" -lt 6 ]; do
  if [ -f /etc/init.d/httpd ]; then break; fi
  sleep 1
  (( i++ )) # built-in arithmetic
done
ls /etc/init.d/httpd

Quote i for protection. Use the break keyword to escape the while loop.

Alternatively, use Bash's built-in arithmetic:
i=0
while (( i < 6 )); do
  if [ -f /etc/init.d/httpd ]; then break; fi
  sleep 1
  (( i++ ))
done
ls /etc/init.d/httpd

This is a good start. Calling test (even as a builtin) in an infinite loop is also wasteful. If you are in bash, you can use the [[ keyword, which has the added benefit of protecting you against unquoted variables in a comparison.

i=0
while (( i < 6 )); do
  if [[ -f /etc/init.d/httpd ]]; then break; fi

  sleep 1
  (( i++ ))
done
ls /etc/init.d/httpd

There is a bug here. ls will run no matter whether the file was found or not.

So now we exit the loop, but how do we notify the caller that it failed? The `break` statement does not return a non-zero. As far as the shell is concerned, the loop completed. We can use a RETVAL variable that we set explicitly to 0 when it succeeds, and 143 to mean "file not found".

i=0
while (( i < 6 )); do
  if [[ -f "$path" ]];then
      RETVAL=0
  else
      RETVAL=143
  fi
    
  [[ "$RETVAL" -eq 0 ]] && break

  sleep 1
  (( i++ ))
done
[[ "$RETVAL" -eq 0 ]] && ls /etc/init.d/httpd || echo "ERR: file never created"

I quote variables in the [[ ]] here even though it's not entirely required.

Which works fine, but we can simplify by simply unsetting RETVAL if there's a success. The test is expressed with [[ -z $xxx ]]. Also it is best practice to send error messages to STDERR (file descriptor --or fd-- #2), using >&2.

i=0
while (( i < 6 )); do
  if [[ -f "$path" ]];then
    unset RETVAL
  else
    RETVAL=143
  fi
    
  [[ -z "$RETVAL" ]] && break

  sleep 1
  (( i++ ))
done
[[ -z "$RETVAL" ]] && ls /etc/init.d/httpd || echo "ERR: file never created" >&2

Now, let's abstract the variables, and set an exit status.

i=0; path="/etc/init.d/httpd"; timeout=6
while (( i < $timeout )); do
  if [[ -f "$path" ]]; then
    unset RETVAL
  else
    RETVAL=143
  fi
    
  [[ -z "$RETVAL" ]] && break

  sleep 1
  (( i++ ))
done
[[ -z "$RETVAL" ]] && ls "$path" || (echo "ERR: file never created" >&2; exit $RETVAL )

Notice that the `exit` is called from a subshell, as if it is called in the current context, it will exit your interactive shell which is annoying and undesirable. There is no way to use return in this context since this is not yet in a function.

Testing with a bogus file.

$ path=/etc/bogusfile
$ while (( i < $timeout )); do
if [[ -f "$path" ]]; then
>   unset RETVAL
> else
>   RETVAL=143
> fi
> [[ -z "$RETVAL" ]] && break
> sleep 1
> (( i++ ))
> done

$ [[ -z "$RETVAL" ]] && ls "$path" || (echo "ERR: file never created"; exit $RETVAL)
ERR: file never created
$ echo $?
143

Now put it in a reusable function!

_testforfile () {
  local i=0
  local path="$1"
  local timeout="$2"

  while (( i < $timeout )); do
    if [[ -f "$path" ]];then
      unset RETVAL
    else
      local RETVAL=143
    fi

    [[ -z "$RETVAL" ]] && break

    sleep 1
    (( i++ ))
  done
  [[ -z "$RETVAL" ]] && ls "$path" || (echo "ERR: file never created" >&2; return "$RETVAL" )
}

Note the use of the `local` keyword so we don't have our custom variables pollute the invoking environment, and the change of exit to return. This function accepts two parameters as input. Call it like so:

_testforfile /etc/init.d/httpd 6

Now, we can enhance this by making sure that at least the second argument is numeric, and even set that to a default value of 5 if it was not provided.

_testforfile () {
  local i=0
  local path="$1"
  local timeout="${2:-5}"

  local re='^[0-9]+$'
  if ! [[ $timeout =~ $re ]]; then
    echo "ERR: Timeout was not a number" >&2
    return 1
  fi

  while (( i < $timeout )); do
    if [[ -f "$path" ]];then
      unset RETVAL
    else
      local RETVAL=143
    fi
    
    [[ -z "$RETVAL" ]] && break
    
    sleep 1
    (( i++ ))
  done
  [[ -z "$RETVAL" ]] && ls "$path" || (echo "ERR: file never created" >&2; return "$RETVAL" )
}

re is a regular expression used in conjunction with the =~ operator.

In action:
$ _testforfile /etc/hosts 47d
ERR: Timeout was not a number


And maybe some tests in a future post.

Thursday, September 22, 2016

Kill an X application when you have multiple X servers running

Mistakenly started a screensaver on my Chromoting session. Needless to say, a real pain when you have to unlock your local workstation AND the remote end after a minute of inactivity.

Of course, now you have duplicate screensavers running, and it's not apparent in the process listing which one belongs to which X server. I certainly don't want to kill the real screensaver on :0, leaving my local console unlocked!

$ ps ax|grep screensave
  6166 ?        S     16:30 xautolock -time 1 -locker xscreensaver-command -lock -detectsleep -corners -+00 -cornerdelay 1
 17200 ?        S      9:25 xscreensaver -nosplash
 25747 ?        S      7:13 xautolock -time 1 -locker xscreensaver-command -lock -detectsleep -corners -+00 -cornerdelay 1
 56143 pts/17   SN+    0:00 grep --color=auto screensave
140723 ?        S      0:07 xscreensaver -nosplash

Seeing as the DISPLAY var is a part of the environment where the X application was invoked, it should be in:

/proc/<PID>/environ

cat /proc/<PID>/environ gives a whole block of unbroken text (unsure why this is not broken), but if you look closely you'll find the DISPLAY variable in there.

The awk command was borrowed from elsewhere on the Internet. It basically says, break things apart using '=' as a field separator (FS) and \0 as a return separator (RS). I tried investigating this further, seeing as '\0' is the null return used in some GNU utils including xargs, went down the following path:

This seemed like the most straightforward way to break it out:
/bin/echo -e $(cat /proc/137572/environ)

However the output was still not broken into lines. Oh well.

Note the use of /bin/echo as echo is usually a shell built-in.

Anyway, let's use awk like I mentioned before, by substituting the PIDs I found above.


PID=17200
$ awk 'BEGIN{FS="="; RS="\0"}  $1=="DISPLAY" {print $2; exit}' /proc/$PID/environ
:0

Well that's not it.

PID=140723
$ awk 'BEGIN{FS="="; RS="\0"}  $1=="DISPLAY" {print $2; exit}' /proc/$PID/environ
:20

Bingo.

Tie it together loosely:
for PID in $(pgrep xscreensa); do echo -n $PID; awk 'BEGIN{FS="="; RS="\0"}  $1=="DISPLAY" {print $2; exit}' /proc/$PID/environ;done
6166:0
25747:20

Now I can quickly tell which processe below to which DISPLAY.


Saturday, September 3, 2016

Simple field searching with MongoDB

Tons of articles detailing full-text search. But that's not what we want.

For us UNIX folks, we just want to "grep" a field:

db.collection.find( {path: /serverpush.html/ } )

The slashes are essentially wildcards.

9menu vs ratmenu vs dmenu

Today I played around with some of the X11 popup menu options available out there, after I had started using i3 as my window manager. i3's simplicity is great, but it also left a bit of a hole in my toolset, as I often used a system-wide context menu, or root menu, to get access to shortcuts, such as.
  1. go into dynamic submenu that contains a list of recent clipboard contents to get them back in the copy-paste buffer
  2. navigate files and folders quickly through submenus
  3. use xfreerdp to connect to the windows machine hostname that I currently have highlighted in another document
  4. call a special paste function that types my clipboard contents (in some remote sessions such as Java-based ones, copy-paste does not work)
  5. take the current X selection and remove http:// and https:// from it (due to Chrome undermining copy and inserting the protocol into copied hostnames

Summary

ratmenu

  • has a "prev" function to recall the previous menu in a hierarchy
  • will not take STDIN or process a script to generate menus
  • no mouse support
  • no hinting that it's a popup (though you can tell your tiling WM to pop it out)
  • easy config

9menu

  • mouse support
  • no text search to select items
  • easy config
  • will take STDIN?
  • window hints for a popover

dmenu

  • full text search to select items
  • no back option, but you can simply call the previous dmenu
  • no mouse support
  • scripts can be used to generate menus
  • a bit tricky to get it to recognize your bashrc

Ideally, there would be a standalone port of the Openbox root menu. However, the code looks very intertwined with Openbox itself. In addition, I find the XML of the Openbox root menu rc.xml to be pretty messy, and would probably re-implement it with a json or plaintext structure.

So, without being able to port the Openbox root menu, the best option for me is 9menu. I've successfully written dynamic menus for it, though nothing too complicated

Errors running iodined server

iodine does not need an actual IPv6 interface, but will fail if AF_INET6 is not enabled, so you should change your kernel params as so:

# sysctl -w net.ipv6.conf.all.disable_ipv6=0

It was likely net.ipv6.conf.all.disable_ipv6=1.

I encountered this problem on Google Compute Engine, which surprisingly does not enable IPv6 by default.

Monday, November 23, 2015

Chrome refuses to "Show in folder" in the correct application on Linux

Trying to set the handler for "inode/directory" was failing to work.


$ xdg-mime default Thunar.desktop inode/directory


Instead, use the interactive mimeopen utility:

$ mimeopen -d $HOME

and choose the application you want.

Friday, January 16, 2015

VPN from a misconfigured cafe using NAT and Linux network namespaces (netns)

Recently I found myself at a cafe that had a wifi connection that was using the whole 10.0.0.0/8 subnet, meaning all addresses from 10.0.0.1-10.255.255.254. This was set up by a professional networking company. In my opinion, someone needs to re-do their CCNA.

So what this means is that if your corporate network is, say, on 10.34.0.0, you will be unable to route traffic easily over the VPN.

I am told there are 2 ways of getting around this

  1. Use network namespaces and NAT'ing to run your chosen applications in their own namespace that is NAT'ed through your real connection
  2. Use iptables prerouting if you know which subnets you are trying to get to on the other side of the VPN.
  3. Convince your coffee shop to use a sane network architecture
I chose #1 for now, and this guide goes over that.

Let's Get Started

Add the network namespace and confirm that it was created:

# ip netns add vpn_nat
# ip netns list

Add virtual ethernet interfaces (peers)
# ip link add name veth0 type veth peer name veth1

Move one of those peers into the vpn_nat namespace
# ip link set veth1 netns vpn_nat

In the namespace context, set up the network
# ip netns exec vpn_nat ifconfig lo up
# ip netns exec vpn_nat ifconfig veth1 192.168.148.2/24 up
# ip netns exec vpn_nat route add default gw 192.168.148.1

The eagle-eyed reader will notice that I am pointing to a gateway that doesn't exist! We fix that like so:
# ifconfig veth0 192.168.148.1/24 up

Test that the vpn_nat namespace can reach veth0

Execute ping in the namespace context vpn_nat:
# ip netns exec vpn_nat ping 192.168.148.1
PING 192.168.148.1 (192.168.148.1) 56(84) bytes of data.
64 bytes from 192.168.148.1: icmp_seq=1 ttl=64 time=0.088 ms
64 bytes from 192.168.148.1: icmp_seq=2 ttl=64 time=0.041 ms

The next step is to connect the veth0 to your physical network either using NAT or bridging. This requires the masquerading kernel module, but I believe it gets loaded automatically.
# sysctl net.ipv4.ip_forward=1
# iptables -t nat -A POSTROUTING -s 192.168.148/24 -d 0.0.0.0/0 -j MASQUERADE

Verify the routing tables

# iptables -t nat -L -n

Ping a google address in the namespace context

#  ip netns exec vpn_nat ping www.google.com

Verify the routing table in the netns

# ip netns exec vpn_natroute

Run your application in the namespace

I am running as an unprivileged user
$  ip netns exec vpn_nat firefox

Undoing

# iptables -t nat -D POSTROUTING 1

References

http://blog.scottlowe.org/2013/09/04/introducing-linux-network-namespaces/
http://how-to.wikia.com/wiki/How_to_set_up_a_NAT_router_on_a_Linux-based_computer
http://www.opencloudblog.com/?p=66