I spent over an hour trying to figure out why libsndfile was claiming that it couldn’t recognize the format of a perfectly good .wav file.

The three disk drives in the array were all Seagate Barracuda ES.2 1TB drives (ST31000340NS).  Two of them still work fine, with all sectors reading without error.  One of them, however, will not do anything but identify its model number.  It doesn’t report the correct drive capacity, and it returns an immediate error to all read or write commands without seeking.

Friends told me about a known failure mode of these drives (and a few other Seagate drives including the Barracuda 7200.11), which can brick the drive.  They said that there was updated firmware from Seagate to fix the problem.  I googled for it, and found a Seagate serial number check utility that will, given a drive serial number, tell you whether you need new firmware.  I entered the serial number of the failed drive, and it reported that there was “No download available for this serial number.”

Further googling, however, turned up a different Seagate page that had specific instructions on how to identify drives subject to the problem, and the firmware download.  Unfortunately Seagate doesn’t provide any release notes on the firmware on their web site, or any detailed description of the failure.  Contrary to what their serial number check utility had told me, my failed drive was in fact one of that needed the upgrade.  The firmware upgrade is provided as an ISO image for a CD.  I downloaded it, booted it, it verified that it was suitable for the drive, and upgraded the firmware.  Unfortunately the drive still didn’t work, and appeared to behave exactly as before.

Further searching revealed that while Seagate doesn’t provide release notes for the firmware on their own web site, they did provide it elsewhere as an “Urgent Field Update” notice to their OEMs, and someone else put it online.  The document gives an overview of the problem, which only occurs at drive power up time.  The drive firmware hits an assertion and goes into a “safe mode”, where safe is defined as bricking the drive.

Seagate says that the firmware upgrade won’t fix the failure once it has already occurred.  If the drive has already failed, it has to be sent back to Seagate for repair.  It would have been rather helpful if Segate’s own web site would have explained that, so that I wouldn’t have wasted time doing a firmware upgrade that wasn’t going to help me at all.

The notice says that Seagate will provide data recovery service for failed drives at no charge.  On the one hand, the drive is under warranty, and I’d rather not spend $160 to replace it.  On the other hand, there is some sensitive data on the drive, so I’d rather not send it to any third party.  This is a good argument for using host-based drive encryption (e.g., LUKS/dm-crypt on Linux).

Instead of sending the drive to Seagate for data recovery, waiting however long that takes, and hoping for the best, I decided that since two of the three drives appeared to be intact, I should be able to do a RAID rebuild myself.  A few years ago when I accidentally wiped out the metadata on a RAID 5, I wrote a program “r5test” to do a rebuild.  At that time, I had an array with all drives intact, so r5test just verified the parity as it did the rebuild.  This time I was actually missing one drive, so I enhanced r5test to allow for a missing drive, in which case it can attempt reconstruction but can’t verify integrity.

With three drives in the array, there are six possible permutations.  The first partition was an 8GB /boot partition, so I tried reconstructing it using each possible permutation.  I found the correct one on the fifth try.  That accomplished, I set it to work reconstructing the entire array onto a single new 2TB drive.  The rate of reconstruction was such that I expected it to take about 25-26 hours.  At some point about 1/3 of the way into the process, r5test exited with no indication of an error.  I don’t know why this happened, but r5test prints a progress message every second, so I was able to restart it from a point slightly before the last stripe it had recovered.

Once r5test had rebuilt the image of the entire array contents, I checked the filesystems.  /tmp was completely trashed, and /var was mostly trashed, but the other filesystems passed fsck with no issues.  I was concerned about /var since it contains MySQL databases.  I was able to find those in lost+found.  I saved the partially recovered /var filesystem, recreated it from a backup, then dropped in the MySQL databases from the rebuilt image.

I took the new drive to the colo to install it in the server.  A second server was physically on top of the one with the failed RAID, so I shut it down and moved it out of the way, in order to make some changes to the SATA cabling in the failed server.  After booting a Fedora recovery CD in order to install GRUB on the new drive, I was able to boot it successfully, about 78 hours after the original failure.

I put the second server back in place, powered it up, and it refused to boot.  The motherboard BIOS displays its information, then the PCI SATA card BIOS displays its information, then the machine resets.  It just went through that cycle continuously, for no obvious reason.  As far as I could tell, it wasn’t even trying to actually boot the disk.  I decided that the best course of action was to install a new motherboard with on-board SATA.  After I got to the computer store, though, I realized that I actually can’t put a new motherboard in that chassis.  It is a 2U chassis with a redundant power supply, but it is from the days of 20-pin ATX power connectors, and all recent motherboards use 24-pin ATX power connectors.  I’ll probably have to replace the whole chassis.  As a temporary measure, I’m replacing the server hardware with a Shuttle X27D mini barebones system (similar to a mini-ITX).  I think the longer-term solution will be to convert it to a virtual machine running on the other server.  It will take about six hours to transfer all the data from the old disk drive to a new on ein the X27D.

In the process of doing all this, I also discovered that my desktop computer at home will not work when I tip it on its side to work on it.  The CPU fan won’t spin, and the motherboard plays a recorded “CPU failed” message.  It works fine when it is upright.  Maybe the fan bearings have worn out.

As a result of all this, I’ve gotten absolutely none of the work done this week that urgently needed to be done.

Later I found that it had succeeded in buffering the whole event. As soon as I dragged the time slider back to the beginning of the event, RealPlayer forgot that it had anything buffered, and started fetching it again from scratch.

Of course, if I could simply download the file that would solve the problem, but they’ve deliberately designed RealPlayer to make that extremely difficult. It’s defective by design.

The first person to invent a car that runs on water…

… may be sitting right in your classroom!”

Wiley’s web site has a PDF of the page.

Apparently Wiley has also put this in one of their calculus textbooks.

This seems right up there with the “energy makes it go” in a grade school book that Feynman complained about.  Don’t the editors of the textbook look at this stuff?  Or do the marketing people get to insert it after the editors are done?

Bleh.

When you boot the machine, Norton 360 pops up a window to start the subscription process, and there is no cancel or quit.  The close item on the menu window is disabled, and the “X” button at the top right doesn’t work.  It won’t let you proceed without creating a new Norton account or entering credentials for an existing account.

Best of all, the fuckwits somehow have managed to subvert the “End program” feature of the Task Manager.  When I try to kill Norton 360, it says “program not responding”, and then the “End now” button does nothing!

Since Vista rearranged everything, it took me a while to figure out where Add/Remove programs went.  Now it’s “Programs and Features” or some such.  It let me uninstall Norton, even while the stupid Norton signup screen was up.

There was also some kind of “BigFix” thing that is supposed to make security updates and fixes easier.  It came up with a message that as of April 30, 2009, emachines customers no longer get BigFix service.

I saved the URL and accepted the agreement, hoping that I’d be able to download the PDF after acceptance, but it still tells me that I don’t have access.

Nothing happened for a few minutes, then the projector stopped just as the trailers finished.  Everyone started talking loudly.  Finally a manager came ito the theatre and started talking, but we couldn’t hear him over the din.  I’m not sure why they didn’t use the P.A. for an announcement.  He was gesturing toward the exit, so everyone filed out.  It was a large theatre, so there was immediately a huge line for the customer service desk.  I told my friends that they could leave their ticket stubs with me and I’d collect the refunds, so that they could take their children home.

The fire department arrived.  They had a hard time getting waiting cars to move out of their way in front of the theatre.  Some people are amazingly stupid.

When I finally got to the head of the line, I found that they were giving refunds along with passes good for future admission.  However, our tickets were purchased on Fandango, and they said that they couldn’t refund those.  I would have gotten upset, but they quickly offered to give me two passes for each ticket stub.

On Friday, though, I very nearly had a toothpaste disaster of my own.  Well, disaster might be too strong a word, but definitely a fiasco.  I had just started using a new tube of toothpaste two days earlier. and when I used it again on Friday I noticed that I’d somehow gotten some toothpaste on the end of the tube, and wondering how I’d managed to do that.  Upon closer inspection, it became apparent that the end of the tube was not sealed at all.  It had been presssed flat, but when I’d squeezed the tube, the toothpaste started squeezing out the end.  If I’d squeezed more vigorously, or if I squeezed the middle of the tube, I would have gotten toothpaste everywhere.

I use Colgate Total Whitening Gel, with which I’ve been quite happy.  I formerly used Aquafresh, and every time I went to the dentist for a cleaning, he or the hygienist would complain about how much plaque I had and how hard it was.  Finally a few years ago the dentist asked if I was using an anti-plaque toothpaste.  I said I was, but he said to switch to Colgate.  Since switching, I’ve had no further problems with plaque.  I’ve very happy with Colgate, other than this unsealed tub, which presumably is the result of a malfunction on the production line.

I use quotation marks for “decides” and “empty” because it’s all part of the printer ink conspiracy.  The cartridge actually still contains more than enough ink to print the few additional pages I need.  The manufacturer has cleverly designed the printer to stop printing when it thinks the cartridge has run low, and doesn’t provide any way that I can force it to continue.  I’d far rather have a manual override and try to get the rest of my pages printed, even if the quality was less than ideal, than just to be completely out of luck.

This time at least it actually was the black ink that ran out.  Last time it was the yellow ink, and the oh-so-clever printer refused to even let me print a black-only document until the yellow ink cartridge was replaced.

Anyone who thinks gasoline is expensive should try computing the cost per gallon of inkjet printer ink from any of the major printer manufacturers, or even from the relatively inexpensive refill suppliers.