[This is a rough draft. There's typos, and some of it may not make sense right now]
On 02/27, our sysadmin installed some software updates to our test webserver (Mac OS X Server, v. 10.4.8). At some point thereafter, the webserver (Apache/WebObjects4.5.1) lost communications with the other app servers in the cluster. Upon investigation, we found that, among other things, the wotaskd process was crashing every 3 minutes.
Here's what /var/log/system.log said:
Mar 16 00:41:19 www-test crashdump[14377]: wotaskd crashed
Mar 16 00:41:19 www-test crashdump[14377]: crash report written to: /Library/Logs/CrashReporter/wotaskd.crash.log
Since I suspected that the software updates had changed some files that wotaskd depended on, I decided to compare the list of libraries loaded by wotaskd to the list of files contained in the software updates that were installed on 02/27.
Among other things, the CrashReporter.log records the libraries loaded by the process before it crashed. The Libraries section of the raw CrashReporter.log looks something like this:
0x9feb8000 - 0x9ff78fff WebObjects /System/Library/Frameworks/WebObjects.framework/Versions/A/WebObjects
0x9ffcd000 - 0x9ffdffff Monitor /System/Library/PrivateFrameworks/Monitor.framework/Versions/A/Monitor
0x9ffea000 - 0x9fff4fff XML /System/Library/PrivateFrameworks/XML.framework/Versions/A/XML
I only needed a list of the file names, so I extracted the list with this command:
tail -32 /Library/Logs/CrashReporter/wotaskd.crash.log | awk '{print $NF}' | sort | tail -31 > libs_sorted.out
tail -32 - give me a list of the last 32 lines in the file (which contained the library files)
| - pipe the output of tail to awk
awk '{print $NF}' - print the last field of each line (in this case the filename)
| sort - pipe the output of awk to sort, and sort the filenames in alphabetical order
tail -31 - clip out some empty lines
> libs_sorted.out - save the resulting list to the file libs_sorted.out
Here's the list of filenames:
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/WebServicesCore.framework/Versions/A/WebServicesCore
/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
/System/Library/Frameworks/EOAccess.framework/Versions/B/EOAccess
/System/Library/Frameworks/EOControl.framework/Versions/B/EOControl
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Security.framework/Versions/A/Security
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/WOExtensions.framework/Versions/A/WOExtensions
/System/Library/Frameworks/WebObjects.framework/Versions/A/WebObjects
/System/Library/PrivateFrameworks/Monitor.framework/Versions/A/Monitor
/System/Library/PrivateFrameworks/MultiScript.framework/Versions/A/MultiScript
/System/Library/PrivateFrameworks/XML.framework/Versions/A/XML
/System/Library/WebObjects/Applications/wotaskd.woa/wotaskd
/usr/lib/dyld
/usr/lib/libSystem.B.dylib
/usr/lib/libauto.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libiconv.2.dylib
/usr/lib/libicucore.A.dylib
/usr/lib/libobjc.A.dylib
/usr/lib/libxml2.2.dylib
/usr/lib/libz.1.dylib
/usr/lib/system/libmathCommon.A.dylib
I then found the list of all files on the system that changed on 20070227 (note that the commands below will be explained in much greater detail a bit later):
touch -t 200702270000 /tmp/date_marker
touch -t 200702280000 /tmp/date_marker2
find / -newer /tmp/date_marker -and \! -cnewer /tmp/date_marker2 -exec ls -d {} \; > 20070227.out
I sorted that list of files:
sort 20070227.out > 20070227_sorted.out
...and found the list of common files:
comm -1 -2 lib_sorted.out 20070227_sorted.out > suspectlibraries.out
The command "comm" displays the common lines between two files.
That gave me this list of files:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
Next, I needed to find out which packages that were installed on 02/27/2007. Whenever you install something using Installer.app (Apple's installer), a receipt package is created in the /Library/Receipts directory. I ran the commands below in the /Library/Receipts directory to get a list of the packages installed on 02/27:
# touch -t 200702270000 /tmp/date_marker
# touch -t 200702280000 /tmp/date_marker2
# find . ! -name . -prune -cnewer /tmp/date_marker ! -cnewer /tmp/date_marker2 -exec ls -dl {} \;
drwxrwxr-x 3 root admin 102 Feb 27 16:34 ./AirPortConfigApps.pkg
drwxrwxr-x 3 root admin 102 Feb 27 16:36 ./DSTUpdateTi-001.pkg
drwxrwxr-x 3 root admin 102 Feb 27 16:35 ./JavaForMacOSX10.4Release5.pkg
drwxrwxr-x 3 root admin 102 Feb 27 16:36 ./SecUpd2006-008Ti.pkg
drwxrwxr-x 3 root admin 102 Feb 27 16:36 ./SecUpd2007-001Ti.pkg
drwxrwxr-x 3 root admin 102 Feb 27 16:36 ./SecUpd2007-002Ti.pkg
drwxrwxr-x 3 root admin 102 Feb 27 16:35 ./WebObjects5.3.3Update.pkg
Here's how to interpret the find command above:
touch -t 200702270000 /tmp/date_marker - creates an empty file named date_marker that's been timestamped exactly midnight on the morning of 02/27/2007.
touch -t 200702280000 /tmp/date_marker2 - create2 an empty file named date_market2 that's been timestamped exactly midnight on the morning of 02/28/2007.
find pathname expression - recursively walk through the directory tree of the pathname , evaluating expression for each file that you find.
. - the current directory
find . - recursively descend down the directory tree, beginning with the current directory
-name pattern - evaluates to true if the last component of the pathname being examined matches pattern
! expression - not operator; it evaluates to true if the expression is false
So the part of the commaned "! -name ." means, "if the name of the file is anything other than "." (the name of the current directory), evaluate to true"
-prune - always evaluates to true. It causes find to not descend into the current file
So this part of the command:
find . ! -name . -prune
literally means:
Recursively walk through the current directory. For every file that is not named "." (the name of the current directory) do not descend any further.
In effect, it means:
"List all the files in the current directory but don't go any deeper."
-cnewer /tmp/date_marker - -cnewer is the same command as -newercm; returns true if the current file has a more recent last change time ("c") or modification time ("m") than /tmp/date_marker
-exec utility [argument...] - execute the utility with the file as an argument (in this case, the command "ls -dl {} \;"; return true if the command returns a zero value as its exit status (meaning there were no errors). If the string ``{}'' appears anywhere in the utility name or the arguments it is replaced by the pathname of the current file.
ls -dl - list the file in long format (l). If the file is a directory (d), list it as a plain file, do not search it recursively
{} - placeholder for the filename
\; - terminate the command; the backslash is to prevent the shell itself from trying to interpret it
Putting it all together, the command above means,
"List all of the files in the current directory (but only in the current directory--don't go any deeper) that were modified between midnight 02/27 and midnight 02/28."
Now I know which packages were installed on 02/27. How do I tell if they installed any of the files that wotaskd depends upon?
Here's how I extracted the filenames that were installed on 02/27. Note that the way I do it below will work for any package, even if it doesn't create a receipts file. However, since all of the packages were "apple-originated", and therefore created receipts, it would've been easier had I used the lsbom utilty (see below). Nonetheless, here goes:
Download the software update disk images from Apple to desktop:
http://www.apple.com/support/downloads/
Mount the disk images
hdid ~/Desktop/DSTUpdateTi-001.dmg
hdid ~/Desktop/SecUpd2007-002Ti.dmg
Use Pacifist to extract the files from the packages to the /tmp directory.
Go into the pkg directory that's created:
cd /tmp/DSTUpdateTi-001 Folder/DSTUpdateTi-001.pkg
Make a list of all files in the packages
find . -exec ls -d '{}' \; -print | sed 's_^.__' | sort > /tmp/AirPortConfigApps.out
find . -exec ls -d '{}' \; -print | sed 's_^.__' | sort > /tmp/DSTUpdateTi-001.out
find . -exec ls -d '{}' \; -print | sed 's_^.__' | sort > /tmp/JavaForMacOSX10.4Release5.out
find . -exec ls -d '{}' \; -print | sed 's_^.__' | sort > /tmp/SecUpd2006-008Ti.out
find . -exec ls -d '{}' \; -print | sed 's_^.__' | sort > /tmp/SecUpd2007-001Ti.out
find . -exec ls -d '{}' \; -print | sed 's_^.__' | sort > /tmp/SecUpd2007-002Ti.out
find . -exec ls -d '{}' \; -print | sed 's_^.__' | sort > /tmp/WebObjects5.3.3Update.out
Compare the files in the packages with the libraries known to have been installed on 02/27:
comm -1 -2 /tmp/suspectlibraries.out /tmp/AirPortConfigApps.out
comm -1 -2 /tmp/suspectlibraries.out /tmp/DSTUpdateTi-001.out
comm -1 -2 /tmp/suspectlibraries.out /tmp/JavaForMacOSX10.4Release5.out
comm -1 -2 /tmp/suspectlibraries.out /tmp/SecUpd2006-008Ti.out
comm -1 -2 /tmp/suspectlibraries.out /tmp/SecUpd2007-001Ti.out
comm -1 -2 /tmp/suspectlibraries.out /tmp/SecUpd2007-002Ti.out
comm -1 -2 /tmp/suspectlibraries.out /tmp/WebObjects5.3.3Update.out
Check that my process for finding packages works, by searching for the filenames in the directories directly:
crasch2:/tmp root# find . -name "CarbonCore" -print
crasch2:/tmp root# find . -name "Metadata" -print
crasch2:/tmp root# find . -name "OSServices" -print
crasch2:/tmp root# find . -name "SearchKit" -print
crasch2:/tmp root# find . -name "Foundation" -print
* Both techniques revealed that only the following file was installed by the packages above:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
Note that I could've skipped the steps of downloading the packages and extracting the list of package names from them. As unsanity notes:
"...A receipt package is created when (and only when) you install something using Installer.app (Apple's installer). First it creates a temporary package based on all the files in the package. Then when you install, it creates the actual package that actually contains a listing of all the files you installed from that package. You can see these two steps if you open Installer.app's Log window and choose "Show everything". The package lists the paths for all the files along with the permissions Installer.app set for them when they were installed (those permissions are part of the actual package in which the files were installed from). The items in /Library/Receipts/ are basically just the shell package without any of the actual files inside. You can see the contents of these by using the lsbom utility. The usage is basically lsbom path/to/archive.bom. Like so:
lsbom /Library/Receipts/MacOSX10.4.pkg/Contents/Archive.bom
This will list all the files that were installed by the package (by absolute path, usually), their installed permissions, file size, and some other information. See the man page for lsbom for more information on the output."
On 02/27, our sysadmin installed some software updates to our test webserver (Mac OS X Server, v. 10.4.8). At some point thereafter, the webserver (Apache/WebObjects4.5.1) lost communications with the other app servers in the cluster. Upon investigation, we found that, among other things, the wotaskd process was crashing every 3 minutes.
Here's what /var/log/system.log said:
Mar 16 00:41:19 www-test crashdump[14377]: wotaskd crashed
Mar 16 00:41:19 www-test crashdump[14377]: crash report written to: /Library/Logs/CrashReporter/wotaskd.crash.log
Since I suspected that the software updates had changed some files that wotaskd depended on, I decided to compare the list of libraries loaded by wotaskd to the list of files contained in the software updates that were installed on 02/27.
Among other things, the CrashReporter.log records the libraries loaded by the process before it crashed. The Libraries section of the raw CrashReporter.log looks something like this:
0x9feb8000 - 0x9ff78fff WebObjects /System/Library/Frameworks/WebObjects.framework/Versions/A/WebObjects
0x9ffcd000 - 0x9ffdffff Monitor /System/Library/PrivateFrameworks/Monitor.framework/Versions/A/Monitor
0x9ffea000 - 0x9fff4fff XML /System/Library/PrivateFrameworks/XML.framework/Versions/A/XML
I only needed a list of the file names, so I extracted the list with this command:
tail -32 /Library/Logs/CrashReporter/wotaskd.crash.log | awk '{print $NF}' | sort | tail -31 > libs_sorted.out
tail -32
| - pipe the output of tail to awk
awk '{print $NF}' - print the last field of each line (in this case the filename)
| sort - pipe the output of awk to sort, and sort the filenames in alphabetical order
tail -31 - clip out some empty lines
> libs_sorted.out - save the resulting list to the file libs_sorted.out
Here's the list of filenames:
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/WebServicesCore.framework/Versions/A/WebServicesCore
/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
/System/Library/Frameworks/EOAccess.framework/Versions/B/EOAccess
/System/Library/Frameworks/EOControl.framework/Versions/B/EOControl
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Security.framework/Versions/A/Security
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/WOExtensions.framework/Versions/A/WOExtensions
/System/Library/Frameworks/WebObjects.framework/Versions/A/WebObjects
/System/Library/PrivateFrameworks/Monitor.framework/Versions/A/Monitor
/System/Library/PrivateFrameworks/MultiScript.framework/Versions/A/MultiScript
/System/Library/PrivateFrameworks/XML.framework/Versions/A/XML
/System/Library/WebObjects/Applications/wotaskd.woa/wotaskd
/usr/lib/dyld
/usr/lib/libSystem.B.dylib
/usr/lib/libauto.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libiconv.2.dylib
/usr/lib/libicucore.A.dylib
/usr/lib/libobjc.A.dylib
/usr/lib/libxml2.2.dylib
/usr/lib/libz.1.dylib
/usr/lib/system/libmathCommon.A.dylib
I then found the list of all files on the system that changed on 20070227 (note that the commands below will be explained in much greater detail a bit later):
touch -t 200702270000 /tmp/date_marker
touch -t 200702280000 /tmp/date_marker2
find / -newer /tmp/date_marker -and \! -cnewer /tmp/date_marker2 -exec ls -d {} \; > 20070227.out
I sorted that list of files:
sort 20070227.out > 20070227_sorted.out
...and found the list of common files:
comm -1 -2 lib_sorted.out 20070227_sorted.out > suspectlibraries.out
The command "comm" displays the common lines between two files.
That gave me this list of files:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
Next, I needed to find out which packages that were installed on 02/27/2007. Whenever you install something using Installer.app (Apple's installer), a receipt package is created in the /Library/Receipts directory. I ran the commands below in the /Library/Receipts directory to get a list of the packages installed on 02/27:
# touch -t 200702270000 /tmp/date_marker
# touch -t 200702280000 /tmp/date_marker2
# find . ! -name . -prune -cnewer /tmp/date_marker ! -cnewer /tmp/date_marker2 -exec ls -dl {} \;
drwxrwxr-x 3 root admin 102 Feb 27 16:34 ./AirPortConfigApps.pkg
drwxrwxr-x 3 root admin 102 Feb 27 16:36 ./DSTUpdateTi-001.pkg
drwxrwxr-x 3 root admin 102 Feb 27 16:35 ./JavaForMacOSX10.4Release5.pkg
drwxrwxr-x 3 root admin 102 Feb 27 16:36 ./SecUpd2006-008Ti.pkg
drwxrwxr-x 3 root admin 102 Feb 27 16:36 ./SecUpd2007-001Ti.pkg
drwxrwxr-x 3 root admin 102 Feb 27 16:36 ./SecUpd2007-002Ti.pkg
drwxrwxr-x 3 root admin 102 Feb 27 16:35 ./WebObjects5.3.3Update.pkg
Here's how to interpret the find command above:
touch -t 200702270000 /tmp/date_marker - creates an empty file named date_marker that's been timestamped exactly midnight on the morning of 02/27/2007.
touch -t 200702280000 /tmp/date_marker2 - create2 an empty file named date_market2 that's been timestamped exactly midnight on the morning of 02/28/2007.
find pathname expression - recursively walk through the directory tree of the pathname , evaluating expression for each file that you find.
. - the current directory
find . - recursively descend down the directory tree, beginning with the current directory
-name pattern - evaluates to true if the last component of the pathname being examined matches pattern
! expression - not operator; it evaluates to true if the expression is false
So the part of the commaned "! -name ." means, "if the name of the file is anything other than "." (the name of the current directory), evaluate to true"
-prune - always evaluates to true. It causes find to not descend into the current file
So this part of the command:
find . ! -name . -prune
literally means:
Recursively walk through the current directory. For every file that is not named "." (the name of the current directory) do not descend any further.
In effect, it means:
"List all the files in the current directory but don't go any deeper."
-cnewer /tmp/date_marker - -cnewer is the same command as -newercm; returns true if the current file has a more recent last change time ("c") or modification time ("m") than /tmp/date_marker
-exec utility [argument...] - execute the utility with the file as an argument (in this case, the command "ls -dl {} \;"; return true if the command returns a zero value as its exit status (meaning there were no errors). If the string ``{}'' appears anywhere in the utility name or the arguments it is replaced by the pathname of the current file.
ls -dl - list the file in long format (l). If the file is a directory (d), list it as a plain file, do not search it recursively
{} - placeholder for the filename
\; - terminate the command; the backslash is to prevent the shell itself from trying to interpret it
Putting it all together, the command above means,
"List all of the files in the current directory (but only in the current directory--don't go any deeper) that were modified between midnight 02/27 and midnight 02/28."
Now I know which packages were installed on 02/27. How do I tell if they installed any of the files that wotaskd depends upon?
Here's how I extracted the filenames that were installed on 02/27. Note that the way I do it below will work for any package, even if it doesn't create a receipts file. However, since all of the packages were "apple-originated", and therefore created receipts, it would've been easier had I used the lsbom utilty (see below). Nonetheless, here goes:
Download the software update disk images from Apple to desktop:
http://www.apple.com/support/downloads/
Mount the disk images
hdid ~/Desktop/DSTUpdateTi-001.dmg
hdid ~/Desktop/SecUpd2007-002Ti.dmg
Use Pacifist to extract the files from the packages to the /tmp directory.
Go into the pkg directory that's created:
cd /tmp/DSTUpdateTi-001 Folder/DSTUpdateTi-001.pkg
Make a list of all files in the packages
find . -exec ls -d '{}' \; -print | sed 's_^.__' | sort > /tmp/AirPortConfigApps.out
find . -exec ls -d '{}' \; -print | sed 's_^.__' | sort > /tmp/DSTUpdateTi-001.out
find . -exec ls -d '{}' \; -print | sed 's_^.__' | sort > /tmp/JavaForMacOSX10.4Release5.out
find . -exec ls -d '{}' \; -print | sed 's_^.__' | sort > /tmp/SecUpd2006-008Ti.out
find . -exec ls -d '{}' \; -print | sed 's_^.__' | sort > /tmp/SecUpd2007-001Ti.out
find . -exec ls -d '{}' \; -print | sed 's_^.__' | sort > /tmp/SecUpd2007-002Ti.out
find . -exec ls -d '{}' \; -print | sed 's_^.__' | sort > /tmp/WebObjects5.3.3Update.out
Compare the files in the packages with the libraries known to have been installed on 02/27:
comm -1 -2 /tmp/suspectlibraries.out /tmp/AirPortConfigApps.out
comm -1 -2 /tmp/suspectlibraries.out /tmp/DSTUpdateTi-001.out
comm -1 -2 /tmp/suspectlibraries.out /tmp/JavaForMacOSX10.4Release5.out
comm -1 -2 /tmp/suspectlibraries.out /tmp/SecUpd2006-008Ti.out
comm -1 -2 /tmp/suspectlibraries.out /tmp/SecUpd2007-001Ti.out
comm -1 -2 /tmp/suspectlibraries.out /tmp/SecUpd2007-002Ti.out
comm -1 -2 /tmp/suspectlibraries.out /tmp/WebObjects5.3.3Update.out
Check that my process for finding packages works, by searching for the filenames in the directories directly:
crasch2:/tmp root# find . -name "CarbonCore" -print
crasch2:/tmp root# find . -name "Metadata" -print
crasch2:/tmp root# find . -name "OSServices" -print
crasch2:/tmp root# find . -name "SearchKit" -print
crasch2:/tmp root# find . -name "Foundation" -print
* Both techniques revealed that only the following file was installed by the packages above:
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
Note that I could've skipped the steps of downloading the packages and extracting the list of package names from them. As unsanity notes:
"...A receipt package is created when (and only when) you install something using Installer.app (Apple's installer). First it creates a temporary package based on all the files in the package. Then when you install, it creates the actual package that actually contains a listing of all the files you installed from that package. You can see these two steps if you open Installer.app's Log window and choose "Show everything". The package lists the paths for all the files along with the permissions Installer.app set for them when they were installed (those permissions are part of the actual package in which the files were installed from). The items in /Library/Receipts/ are basically just the shell package without any of the actual files inside. You can see the contents of these by using the lsbom utility. The usage is basically lsbom path/to/archive.bom. Like so:
lsbom /Library/Receipts/MacOSX10.4.pkg/Contents/Archive.bom
This will list all the files that were installed by the package (by absolute path, usually), their installed permissions, file size, and some other information. See the man page for lsbom for more information on the output."
no subject
Date: 2007-03-16 04:18 pm (UTC)