Friday, November 12, 2010

Stopping whichs

I was using a tool the other day, and I started it in the background. It didn't come up and, when I looked it had stopped. When this has happened in the past I foreground it and it continues working. I've only noticed this on rare occasions, and I'd previously put it down to some misconfiguration of the system. However, one of my colleagues had also noticed it, so this was the ideal opportunity to figure out what really was going on.

The first step was to identify which process was stopped using jobs -l:

$ jobs -l
[1]- 25195 Running                 process1 &
[2]+ 25223 Stopped (tty output)    process2

Having done that, the next step was to find out where the process had actually stopped. This information can be obtained using ptree which prints out the process call tree:

$ ptree 25223
511   /usr/lib/ssh/sshd
   25160 /usr/lib/ssh/sshd
     25161 /usr/lib/ssh/sshd
       25166 -bash
         25223 /bin/sh process2
           25232 sed -n $p
             25233 /usr/bin/csh -f /usr/bin/which java java
               25234 /usr/bin/stty erase ^H 

So the process has stalled in stty setting the erase character to be ^H. The callstack, printed by pstack, was not very enlightening.

$ pstack 25234
25234:  /usr/bin/stty erase ^H
  feef14d7 ioctl    (0, 540f, 8067988)
  080516f8 main     (3, 8047b24, 8047b34, 80511ff) + 40c
  0805125d _start   (3, 8047c08, 8047c16, 8047c1c, 0, 8047c1f) + 7d 

However the interesting step is from which to stty. What's interesting about which is that it is a C-shell script. The interesting bit is the following:

#! /usr/bin/csh -f
if ( -r ~/.cshrc && -f ~/.cshrc ) source ~/.cshrc

So which sources the .cshrc file, and my .cshrc file happened to contain stty erase ^H. So why does this cause the process to stop?

Well stty controls the characteristics of the terminal, but when the script is executing in the background, there is no terminal. When there's no terminal, stty stops and waits for one to appear!

The easiest is to move the call to stty into my .login file. The .login file is only parsed at login, and not every time a shell is started. Alternatively, it's possible to check for the existence of a prompt:

if ($?prompt) then
  if ("$prompt" =~ ?*) then
  /usr/bin/stty  erase ^H