AIX getcwd() weirdness

When I came into the office my colleague immediately confronted me with a weird issue. It seems all of our applications stopped working due to an issue with a NFS server. Almost all commands and utilities hang and spawned NFS timeouts after a while.

The most simple explanation would be a dependency on the NFS share. But I knew this couldn’t be the case since the NFS is only used to transfer some data, but not by any apllication. At least, it shouldn’t.

It was quite simple to reproduce this issue on a test machine. I mounted a test NFS share and firewalled it afterwards to simulate a failed NFS server. Example commands that fail are: “which”, “pwd”, “enq”, etc.

Time to analyze what’s going on:

$ truss -f pwd

... cut ...

156966: getdirent64(3, 0x20002758, 4096) = 3072
156966: statx("./../../.", 0x2FF210B8, 176, 021) = 0
156966: statx("./../../..", 0x2FF210B8, 176, 021) = 0
156966: statx("./../../home", 0x2FF210B8, 176, 021) = 0
156966: statx("./../../fci", 0x2FF210B8, 176, 021) = 0
156966: statx("./../../lib", 0x2FF210B8, 176, 021) = 0
156966: statx("./../../libmgr", 0x2FF210B8, 176, 021) = 0
156966: statx("./../../opt", 0x2FF210B8, 176, 021) = 0
156966: statx("./../../DBDEV", 0x2FF210B8, 176, 021) = 0
156966: statx("./../../tmp", 0x2FF210B8, 176, 021) = 0
156966: statx("./../../usr", 0x2FF210B8, 176, 021) = 0
156966: statx("./../../var", 0x2FF210B8, 176, 021) = 0
156966: statx("./../../dev", 0x2FF210B8, 176, 021) = 0

... cut ...

statx("./../../DDR", 0x2FF210B8, 176, 021) = 0

 

Now I know there is a lot of crap in / that really shouldn’t be there… but this is how the company has setup their systems in the past.

So what’s going on here? All command’s like pwd, which, enq, etc. execute a statx() on almost all entries in the root filesystem. Since /DDR is a dead NFS mount, the application will wait for statx() on /DDR indefinitely.

But why would these commands cause a statx() on almost all entries in the root filesystem? It doesn’t make sense, introduces a performance penalty and a dead NFS share in the root filesystem even prevents applications from working properly.

Some further investigation, with the dead NFS mount being unmounted, showed actually that the statx() stops when the first entry of the current path is reached. To illustrate this, below is the output of “truss -t statx pwd” executed both from /usr and /var.

$ cd /var
$ truss -t statx pwd
statx(0xF01E7D30, 0x2FF20EB0, 76, 0) = 0
statx(0xF01E7D34, 0x2FF21018, 176, 020) = 0
statx("./", 0x2FF21018, 176, 020) = 0
statx("./../", 0x2FF20EB0, 76, 0) = 0
statx("./../.", 0x2FF210C8, 176, 021) = 0
statx("./../..", 0x2FF210C8, 176, 021) = 0
statx("./../home", 0x2FF210C8, 176, 021) = 0
statx("./../fci", 0x2FF210C8, 176, 021) = 0
statx("./../lib", 0x2FF210C8, 176, 021) = 0
statx("./../libmgr", 0x2FF210C8, 176, 021) = 0
statx("./../opt", 0x2FF210C8, 176, 021) = 0
statx("./../DBDEV", 0x2FF210C8, 176, 021) = 0
statx("./../tmp", 0x2FF210C8, 176, 021) = 0
statx("./../usr", 0x2FF210C8, 176, 021) = 0
statx("./../var", 0x2FF210C8, 176, 021) = 0
/var

 

$ cd /usr
$ truss -t statx pwd
statx(0xF01E7D30, 0x2FF20EB0, 76, 0) = 0
statx(0xF01E7D34, 0x2FF21018, 176, 020) = 0
statx("./", 0x2FF21018, 176, 020) = 0
statx("./../", 0x2FF20EB0, 76, 0) = 0
statx("./../.", 0x2FF210C8, 176, 021) = 0
statx("./../..", 0x2FF210C8, 176, 021) = 0
statx("./../home", 0x2FF210C8, 176, 021) = 0
statx("./../fci", 0x2FF210C8, 176, 021) = 0
statx("./../lib", 0x2FF210C8, 176, 021) = 0
statx("./../libmgr", 0x2FF210C8, 176, 021) = 0
statx("./../opt", 0x2FF210C8, 176, 021) = 0
statx("./../DBDEV", 0x2FF210C8, 176, 021) = 0
statx("./../tmp", 0x2FF210C8, 176, 021) = 0
statx("./../usr", 0x2FF210C8, 176, 021) = 0
/usr

The environment I used was stripped and people in #aix on freenode were able to reproduce this behaviour on AIX 5.1, 5.2 and 5.3 systems.

Since this looked like a bug I opened a support case with IBM. The next day IBM replied with APAR “IZ79889” . This APAR states:

The performance of getwd is degraded because 
it performs a statx on all directories up to /
in order to find the pathname. If crossing a 
mount point, it does a statx on all directories 
in order to find the inode number that matches 
the mount point.

So it seems getwd/getcwd performs a statx() on all directories in order to find the pathname and the inode number that matches the mountpoint.

It’s needless to say that this is very, very bad 😉

IBM fixed this in later releases.

 

 

 

 

This entry was posted in Uncategorized. Bookmark the permalink.

Comments are closed.