Problem:
This time in one of my scripts I had to create a replicated filesystem-like structure (whole directory tree with selected files) using symbolic links. One of the problems with managing this kind of structure was maintaining an appropriate resolution for all the symlinks throughout the whole update process of the replicated tree, which occurred occasionally. Preferably atomic rename would do the trick. And as Linux ‘mv’ is nothing more than a nice wrapper around rename call, doing this right should not have been hard at all.
In fact, a simple problem occurred not where I would suspect. I couldn’t force the ‘mv’ command to rename a symlink if the destination had already existed as a symbolic link to another directory.
For instance, let’s assume the following structure:
./foo -> ./baz (foo being a symbolic link to directory baz)
./bar -> ./qux (bar being a symbolic link to directory qux)
Then executing:
mv foo bar
would in fact move the foo symlink under qux directory (symlinked as bar).
Though this seemed a bit absurd, I could not find a way to make it work as desired (forcing an overwrite of bar with foo).
Solution:
As always, the most trivial solution was the one I hadn’t thought about. Condemned once again for working on archaic systems, I found out that quite a long time ago (but later than the installation of that particular system) an option precisely dedicated for this problem was added to the ‘mv’ command.
All the recent versions of ‘mv’ support the ‘-T’ or ‘–no-target-directory’ switch. It disables treating the last operand in a special way when it’s a directory/symlink to a directory.
So doing:
mv -f -T foo bar
would work as expected. Though I’m not 100% sure about its atomicity (according to POSIX it should be), it’s the best I could get. And I haven’t noted any problems/race conditions so far.
More about the idea behind ‘Target directory’ behaviour can be found in the GNU Coreutils manuals.
Last but not least, always check if you’re working on the latest version of your tools. Or at least if updating won’t fix your issue right away.
Great info and just at the time I was looking for it.
Unfortunately, I’m working on a system that can’t be updated to the version of GNU mv with the -T option.
So, how would you go about an atomic directory replacement without this option ?
Thanks
I don’t know how it looks by your side nor what’s the underlying purpose of atomic directory replacement, but the very first thing that comes to my mind is just overriding the standard system ‘mv’ command with a userland updated version.
Try downloading an updated version to your ~/bin directory (I don’t know how the paths are set on your system) or just invoke it directly from wherever you want. If it’s not feasible, try using ‘rename’ command. If still that’s not possible, then I think some ‘rename’ system call wrapper should be used.
I suppose these 2 links could shed some light on the issue:
http://blog.moertel.com/articles/2005/08/22/how-to-change-symlinks-atomically
http://www.weirdnet.nl/apple/rename.html
The rename man page notes that in case of an overwrite there can be a window where both files exist and point to the same place. So the operation as a whole is not atomic, but the replacement of the destination (under Linux with a sane filesystem) should be. I can imagine that different filesystems could break this.
I would certainly test this well before relying on it. Note that this post claims that rename is not atomic on OS X (and also provides a handy test script):
http://www.weirdnet.nl/apple/rename.html
Thanks for the input.
I’m sorry, I was thinking mostly of Linux single filesystem operations. In that case according to the specification (http://www.opengroup.org/onlinepubs/9699919799/functions/rename.html) it should behave in this manner:
“If the link named by the new argument exists, it shall be removed and old renamed to new. In this case, a link named new shall remain visible to other processes throughout the renaming operation and refer either to the file referred to by new or old before the operation began”.
And that was the kind of atomicity I needed. A very specific structure being a poor man’s userland imitation of a subset of UnionFS functionality. I only needed that the directories (symlinks) remain accessible throughout the process. And it worked OK for quite a time. Though I should note it was a kind of workaround you would use for a private project. For production use using proper full-blown union filesystem would be much more reliable, faster and easier.
Good point about OS X – I forgot to emphasise it, as I’m rather too much Linux-oriented. ;-)
lets say you created version 20 in package-20 and version 21 in package-21
You now need to flip the link package -> package-20 to package -> package-21
This command does that with the ln command, without using mv for places where there is no mv special for this purpose.
ln –symbolic –force –no-dereference package-21 package
more concisely,
ln -sfn /package-21 /package
to quote the manual regarding no-dereference: treat destination that is a symlink to a directory as if it were a normal file
David
Davids approach works in principle, but still isn’t atomic:
$ strace ln -sfn package-21 package
…
symlink(“package-21″, “package”) = -1 EEXIST (File exists)
unlink(“package”) = 0
symlink(“package-21″, “package”) = 0
There is a very small window in between unlink() and symlink() where “package” won’t exist. At our company we have a use case where this window is large enough to break some automatic processes. Thus, we now use an approach combining both the “ln” and “mv” behaviours:
$ ln -sfn package-21 package.tmp
$ strace mv -f -T package.tmp package
…
rename(“package.tmp”, “package”) = 0
This will create a symlink called package.tmp (replacing any current package.tmp symlink if one happens to lie around), then atomically replace the package symlink with the new one.