Showing posts with label linux. Show all posts
Showing posts with label linux. Show all posts

Thursday, September 29, 2016

Be more intelligent with your `sleep`--or-- how to overengineer your scripts

The universal solution to waiting for something to be ready in shell scripting is the `sleep` command.

Here, we're waiting for a dir to be created (say from a `yum install httpd` going on in another terminal), so we can `ls` the contents, perhaps as part of a script that configures httpd.

sleep 6
ls /etc/init.d/httpd

But is there a better way? What if the directory exists almost immediately? You've wasted nearly 6 seconds unnecessarily, which if you do this a lot in your script, adds a bunch of time.

We need a way to continuously query if the resource exists. And it needs to be with a command that sets an error exit code if the resource is not found. Let's use `stat` to do this.

Leveraging the fact that `stat` will set a non-zero exit code on failure

while true; do
  stat /etc/init.d/httpd
  if [ $? -eq 0 ]; then # check if the stat was successful
    break
  fi
done
ls /etc/init.d/httpd

'true' is a command, whose output is nothing and whose exit status is 0. In terms of performance we'd probably like not to call a binary, but `true` is a shell built-in. The no-op operator `:` accomplishes the same thing, ex: `while :; do`.

We can simplify the if statement by using the && operator which executes the following command break, if stat exits without error (sets a 0 status)

while true; do
  stat /etc/init.d/httpd && break
done
ls /etc/init.d/httpd

Instead of stat, we can use the `test` command (aliased as `[`). Here we check for a file using -f.

while true; do
  if [ -f /etc/init.d/httpd ]; then break; fi
done
ls /etc/init.d/httpd

However, in these examples, if the file never exists, the loop will never exit.

So, instead we can define a timeout and a shorter `sleep` interval, and a counter (i) to track the iterations:

i=0
while [ "$i" -lt 6 ]; do
  if [ -f /etc/init.d/httpd ]; then break; fi
  sleep 1
  (( i++ )) # built-in arithmetic
done
ls /etc/init.d/httpd

Quote i for protection. Use the break keyword to escape the while loop.

Alternatively, use Bash's built-in arithmetic:
i=0
while (( i < 6 )); do
  if [ -f /etc/init.d/httpd ]; then break; fi
  sleep 1
  (( i++ ))
done
ls /etc/init.d/httpd

This is a good start. Calling test (even as a builtin) in an infinite loop is also wasteful. If you are in bash, you can use the [[ keyword, which has the added benefit of protecting you against unquoted variables in a comparison.

i=0
while (( i < 6 )); do
  if [[ -f /etc/init.d/httpd ]]; then break; fi

  sleep 1
  (( i++ ))
done
ls /etc/init.d/httpd

There is a bug here. ls will run no matter whether the file was found or not.

So now we exit the loop, but how do we notify the caller that it failed? The `break` statement does not return a non-zero. As far as the shell is concerned, the loop completed. We can use a RETVAL variable that we set explicitly to 0 when it succeeds, and 143 to mean "file not found".

i=0
while (( i < 6 )); do
  if [[ -f "$path" ]];then
      RETVAL=0
  else
      RETVAL=143
  fi
    
  [[ "$RETVAL" -eq 0 ]] && break

  sleep 1
  (( i++ ))
done
[[ "$RETVAL" -eq 0 ]] && ls /etc/init.d/httpd || echo "ERR: file never created"

I quote variables in the [[ ]] here even though it's not actually required.

Which works fine, but we can simplify by simply unsetting RETVAL if there's a success. The test is expressed with [[ -z $xxx ]]. Also it is best practice to send error messages to STDERR (file descriptor --or fd-- #2), using >&2.

i=0
while (( i < 6 )); do
  if [[ -f "$path" ]];then
    unset RETVAL
  else
    RETVAL=143
  fi
    
  [[ -z "$RETVAL" ]] && break

  sleep 1
  (( i++ ))
done
[[ -z "$RETVAL" ]] && ls /etc/init.d/httpd || echo "ERR: file never created" >&2

Now, let's abstract the variables, and set an exit status.

i=0; path="/etc/init.d/httpd"; timeout=6
while (( i < $timeout )); do
  if [[ -f "$path" ]]; then
    unset RETVAL
  else
    RETVAL=143
  fi
    
  [[ -z "$RETVAL" ]] && break

  sleep 1
  (( i++ ))
done
[[ -z "$RETVAL" ]] && ls "$path" || (echo "ERR: file never created" >&2; exit $RETVAL )

Notice that the `exit` is called from a subshell, as if it is called in the current context, it will exit your interactive shell which is annoying and undesirable. There is no way to use return in this context since this is not yet in a function.

Testing with a bogus file.

$ path=/etc/bogusfile
$ while (( i < $timeout )); do
if [[ -f "$path" ]]; then
>   unset RETVAL
> else
>   RETVAL=143
> fi
> [[ -z "$RETVAL" ]] && break
> sleep 1
> (( i++ ))
> done

$ [[ -z "$RETVAL" ]] && ls "$path" || (echo "ERR: file never created"; exit $RETVAL)
ERR: file never created
$ echo $?
143

Now put it in a reusable function!

_testforfile () {
  local i=0
  local timeout="$1"
  local path="$2"

  while (( i < $timeout )); do
    if [[ -f "$path" ]];then
      unset RETVAL
    else
      local RETVAL=143
    fi

    [[ -z "$RETVAL" ]] && break

    sleep 1
    (( i++ ))
  done
  [[ -z "$RETVAL" ]] && ls "$path" || (echo "ERR: file never created" >&2; return "$RETVAL" )
}

Note the use of the `local` keyword so we don't have our custom variables pollute the invoking environment, and the change of exit to return. This function accepts two parameters as input. Call it like so:

_testforfile 6 /etc/init.d/httpd

Now, we can enhance this by making sure that at least the first argument is numeric, and even set that to a default value of 5 if it was not provided.

_testforfile () {
  local i=0
  local timeout="${2:-5}"
  local path="$1"

  local re='^[0-9]+$'
  if ! [[ $timeout =~ $re ]]; then
    echo "ERR: Timeout was not a number" >&2
    return 1
  fi

  while (( i < $timeout )); do
    if [[ -f "$path" ]];then
      unset RETVAL
    else
      local RETVAL=143
    fi
    
    [[ -z "$RETVAL" ]] && break
    
    sleep 1
    (( i++ ))
  done
  [[ -z "$RETVAL" ]] && ls "$path" || (echo "ERR: file never created" >&2; return "$RETVAL" )
}

re is a regular expression used in conjunction with the =~ operator.

In action:
$ _testforfile 47d /etc/hosts
ERR: Timeout was not a number

And maybe some tests in a future post.

Thursday, September 22, 2016

Kill an X application when you have multiple X servers running

Mistakenly started a screensaver on my Chromoting session. Needless to say, a real pain when you have to unlock your local workstation AND the remote end after a minute of inactivity.

Of course, now you have duplicate screensavers running, and it's not apparent in the process listing which one belongs to which X server. I certainly don't want to kill the real screensaver on :0, leaving my local console unlocked!

$ ps ax|grep screensave
  6166 ?        S     16:30 xautolock -time 1 -locker xscreensaver-command -lock -detectsleep -corners -+00 -cornerdelay 1
 17200 ?        S      9:25 xscreensaver -nosplash
 25747 ?        S      7:13 xautolock -time 1 -locker xscreensaver-command -lock -detectsleep -corners -+00 -cornerdelay 1
 56143 pts/17   SN+    0:00 grep --color=auto screensave
140723 ?        S      0:07 xscreensaver -nosplash

Seeing as the DISPLAY var is a part of the environment where the X application was invoked, it should be in:

/proc/<PID>/environ

cat /proc/<PID>/environ gives a whole block of unbroken text (unsure why this is not broken), but if you look closely you'll find the DISPLAY variable in there.

The awk command was borrowed from elsewhere on the Internet. It basically says, break things apart using '=' as a field separator (FS) and \0 as a return separator (RS). I tried investigating this further, seeing as '\0' is the null return used in some GNU utils including xargs, went down the following path:

This seemed like the most straightforward way to break it out:
/bin/echo -e $(cat /proc/137572/environ)

However the output was still not broken into lines. Oh well.

Note the use of /bin/echo as echo is usually a shell built-in.

Anyway, let's use awk like I mentioned before, by substituting the PIDs I found above.


PID=17200
$ awk 'BEGIN{FS="="; RS="\0"}  $1=="DISPLAY" {print $2; exit}' /proc/$PID/environ
:0

Well that's not it.

PID=140723
$ awk 'BEGIN{FS="="; RS="\0"}  $1=="DISPLAY" {print $2; exit}' /proc/$PID/environ
:20

Bingo.

Tie it together loosely:
for PID in $(pgrep xscreensa); do echo -n $PID; awk 'BEGIN{FS="="; RS="\0"}  $1=="DISPLAY" {print $2; exit}' /proc/$PID/environ;done
6166:0
25747:20

Now I can quickly tell which processe below to which DISPLAY.


Saturday, September 3, 2016

Errors running iodined server

iodine does not need an actual IPv6 interface, but will fail if AF_INET6 is not enabled, so you should change your kernel params as so:

# sysctl -w net.ipv6.conf.all.disable_ipv6=0

It was likely net.ipv6.conf.all.disable_ipv6=1.

I encountered this problem on Google Compute Engine, which surprisingly does not enable IPv6 by default.

Friday, January 16, 2015

VPN from a misconfigured cafe using NAT and Linux network namespaces (netns)

Recently I found myself at a cafe that had a wifi connection that was using the whole 10.0.0.0/8 subnet, meaning all addresses from 10.0.0.1-10.255.255.254. This was set up by a professional networking company. In my opinion, someone needs to re-do their CCNA.

So what this means is that if your corporate network is, say, on 10.34.0.0, you will be unable to route traffic easily over the VPN.

I am told there are 2 ways of getting around this

  1. Use network namespaces and NAT'ing to run your chosen applications in their own namespace that is NAT'ed through your real connection
  2. Use iptables prerouting if you know which subnets you are trying to get to on the other side of the VPN.
  3. Convince your coffee shop to use a sane network architecture
I chose #1 for now, and this guide goes over that.

Let's Get Started

Add the network namespace and confirm that it was created:

# ip netns add vpn_nat
# ip netns list

Add virtual ethernet interfaces (peers)
# ip link add name veth0 type veth peer name veth1

Move one of those peers into the vpn_nat namespace
# ip link set veth1 netns vpn_nat

In the namespace context, set up the network
# ip netns exec vpn_nat ifconfig lo up
# ip netns exec vpn_nat ifconfig veth1 192.168.148.2/24 up
# ip netns exec vpn_nat route add default gw 192.168.148.1

The eagle-eyed reader will notice that I am pointing to a gateway that doesn't exist! We fix that like so:
# ifconfig veth0 192.168.148.1/24 up

Test that the vpn_nat namespace can reach veth0

Execute ping in the namespace context vpn_nat:
# ip netns exec vpn_nat ping 192.168.148.1
PING 192.168.148.1 (192.168.148.1) 56(84) bytes of data.
64 bytes from 192.168.148.1: icmp_seq=1 ttl=64 time=0.088 ms
64 bytes from 192.168.148.1: icmp_seq=2 ttl=64 time=0.041 ms

The next step is to connect the veth0 to your physical network either using NAT or bridging. This requires the masquerading kernel module, but I believe it gets loaded automatically.
# sysctl net.ipv4.ip_forward=1
# iptables -t nat -A POSTROUTING -s 192.168.148/24 -d 0.0.0.0/0 -j MASQUERADE

Verify the routing tables

# iptables -t nat -L -n

Ping a google address in the namespace context

#  ip netns exec vpn_nat ping www.google.com

Verify the routing table in the netns

# ip netns exec vpn_natroute

Run your application in the namespace

I am running as an unprivileged user
$  ip netns exec vpn_nat firefox

Undoing

# iptables -t nat -D POSTROUTING 1

References

http://blog.scottlowe.org/2013/09/04/introducing-linux-network-namespaces/
http://how-to.wikia.com/wiki/How_to_set_up_a_NAT_router_on_a_Linux-based_computer
http://www.opencloudblog.com/?p=66

Thursday, January 8, 2015

Recovering Tinder messages from users who've deleted their accounts

Okay, I've used Tinder.

However, we'll put that aside and get more technical here.

I ran into the situation where a person had given me some info to add her on Facebook, so that we could continue the conversation, but then deleted/disabled her Tinder account!

For those who don't know, Tinder will remove such a person from your matches and you cannot look at the conversation history anymore. Well, that sucks!

So here's a really, really, technical solution.

For security reasons, Android will not let you look in /data/com.tinder. Inside /data/com.tinder/db/tinder.db is a copy of all of the messages you've sent and received in the app. So we need this file, but without a rooted phone, the only way to access this data is to access it through Tinder, which would require us to modify and recompile Tinder ;-). If you try to use a file manager to browse /data, nothing will show up.

At this point you can root your phone, which requires unlocking the bootloader, which has been claimed to void your warranty, and naturally, when you unlock the bootloader the phone is wiped for security reasons.

So rooting is really not an option for such a silly little thing. You might as well give up on this potential person, even if they are your soul mate? :-(

The workaround is to enable ADB USB debugging in Android, and take a backup using the adb tool on your computer, while it's connected to the handset. Do not set a password!

adb backup -f my_backup.ab -apk -shared -all

And confirm from the phone that you want to allow this backup.

At this point you'll have a huge file called my_backup.ab. You'll need to extract this using the Android Backup Extractor which will require Java7 to be installed. Then you can run:

java -jar abe.jar unpack my_backup.ab my_backup.tar

Which will create a standard .tar file called my_backup.tar.

We can then extract this using tar

tar -xf my_backup.tar

Then navigate to the folder that's created, apps/ then com.tinder/db

From here, open the database (a SQLite DB) using sqlite3 command:

sqlite3 tinder.db

Let's view some messages!

select * from main.messages LIMIT 10;

Anything look familiar here? It should!

Okay so this is all out of order. What we are looking for are the most recent conversations. The field we are interested in is called 'created', so let's order messages by that!

SELECT * from main.messages ORDER BY created ASC;

54ef7ac30147af1b7d000192|53ef05f9f6061a2f32055da4||2014-02-08T19:28:06.922Z||Lol I am, do you want to add me on Facebook? Suzy Queens|

The names and unique IDs have been changed to protect everyone involved. But hey! There's the message I've been looking for! A match made in heaven. The person will owe you a drink for all of the hard work you've done ;-)

So my advice would be, if you get a phone number or last name from someone you are interested in...take a screenshot right then and there...Tinder will hide the conversation if that person then leaves Tinder or unmatches you. Otherwise, use this method if you are nerdy, or just can't take a hint. ;-)

Saturday, April 5, 2014

Linux don't care...

Say for example, Windows won't let you delete or back up an open file. Even if you wanted to easily create a snapshot of a volume and back up an open file in a consistent manner, it's not that easy.

Linux don't care...

In fact, a recent question I was asked in an interview pre-screening, was whether you can delete an in-use logfile and if that space is automatically marked free by the OS?

The answer to that is, no. On Linux, a file can be deleted when it's in-use. The file will disappear from the FS, but the handle to the file will remain open, and in fact, the application will continue to write to the file even when its hard link count is now exactly zero! That log file will actually appear in the /proc filesystem under that process' PID, in its list of open files. In order to actually delete the file you need to close the file handle. This is done in various ways depending on the application, but it can usually be triggered quite easily, and as a sysadmin, you should why and how you should do this.

UNIX expects you to be a competent and knowledgeable sysadmin, and this has the advantage of creating a very predictable environment with few exceptions to basic tenets laid out as a contract of sorts

I like that.

I like being treated like an adult.