Blog of Annoying Technical Details

I am a huge fan of archive.org, it is a very useful website, and a extremely important mission. Recently, I was enjoying the Wayback Machine when trying to find some information on iPodLinux. It turns out that ipodlinux.org has been down for a long time, perhaps no one pays the hosting bills anymore. But thankfully, it was all archived by archive.org.

So I needed to get some source files like Pdpod.c.gz from ipodlinux.org/Pdpod, which was last archived on 2007-12-09. The files are there, but the downloading behaved strangely, like it ended too quickly or something. Then the resulting files were always corrupt. So I tried wget, it kept on retrying the download, saying Connection closed at byte 6372. Retrying.

I decided to look at the binary file with less to see if I could deduce anything. Eureka! I can see it, but it still doesn’t uncompress, gunzip says gzip: Pdpod.c.gz: unexpected end of file. Hmm, but less can see it, so gzip must work somehow. I ran it using -c to send the uncompressed data to stdout on the fly:

gunzip -c Pdpod.c.gz > Pdpod.c

And even though gzip still complains, the whole file is there. Yeeee haw…

So I have an SD card formatted FAT32 for my Palm TX. I want to use rsync to sync mp3s, images, etc. to that card from Mac OS X, since Missing Sync has royally sucked and wasted lots of my time trying to do the same thing. I wrote my handy little script using rsync, but I noticed that it was copying all of the files each time. Strange…

It turns out that FAT32 has a very low time resolution, compared to ext2/3 or HFS+. Luckily, rsync has an option to solve that. So I added --modify-window=1 and it works now.

Joy of joys, my server is crashes for some ungodly reason, and it took out my MySQL server. Arg… The damn thing won’t start at all, it crashes badly everytime I start it. And it gives me a big, fat crash message, and even the backtrace is broken:

InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
080811 15:51:19  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 3 2057517825.
InnoDB: Doing recovery: scanned up to log sequence number 3 2057578233
080811 15:51:19  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_connections=100
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 217599 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xb511d4a8, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81c06b9
0x83cba12
0x83cc69b
0x8390215
0x8392120
0x837f38a
0x83aa73c
0x82f0308
0xb7ed10bd
0xb7d0c01e
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

But thankfully, this MySQL was running on Debian. So through the magic of apt, I got everything running again. Basically, I renamed the key folders, then I purged MySQL server and reinstalled it, even deleting the databases (since they were moved elsewhere anyway):


mv /var/lib/mysql /var/lib/mysql-old
mv /var/log/mysql /var/log/mysql-old
aptitude purge mysql-server mysql-server-5.0
aptitude install mysql-server


Ok, now at least mysqld starts. Then I switched back the original files, keeping the Debian generated one, just in case:


/etc/init.d/mysql stop
mv /var/lib/mysql /var/lib/mysql-deb
mv /var/log/mysql /var/log/mysql-deb
mv /var/lib/mysql-old /var/lib/mysql
mv /var/log/mysql-old /var/log/mysql
/etc/init.d/mysql start


Wow, it worked… I wonder how borked the data is, it’s time for a backup, then we’ll see…. ah, scheiße, there is more recover pain to endure. Hmm… first one is a simple one. I tried deleting some cruft databases, but it wasn’t fully deleting them. And none of the databases were showing up in phpMyAdmin or SHOW DATABASES;. Turns out the perms were wrong. This helped that problem:


chown mysql.mysql /var/lib/mysql/


Ok, things were going swimmingly, until I got an error #1 - Can't create/write to file '/tmp/#sql_3c2e_0.MYI' (Errcode: 13). Crazy ass shit. Turns out that /tmp had strange permissions, only root can write:


root@blinky:mysql > ls -ld /tmp/
drwxr-xr-x 17 root root 262144 2008-08-11 18:19 /tmp//


So I rectified that with a little chmod 1777 /tmp/, but now, WTF!??!, mysql crashes again on startup, with the same error!

So I am using Ubuntu Hardy more and more, and am quite impressed at how integrated things are. For a long time, Debian has been very well integrated on the command line for servers and things like that, but Ubuntu is really tackling giving all of the myriad apps a standard GUI. GNOME is helping a lot too, of course.

So I just set up a firewire drive to use for development work, and I saw in the Properties panel for the disk, under the Volume tab, I could set the mountpoint name. So I typed out a whole path in the Mount Point: field, /home/hans/code/openembedded, as I usually would when mounting on the command line. Then hit Close, and unmounted and remounted the disk.

When I plugged it back it, I got this obtuse message:

Cannot mount volume.
Unable to mount the volume.
Details
mount_point cannot contain the following characters: newline, G_DIR_SEPARATOR (usually /)

Hmm, WTF? Now what, the disk won’t mount, so I can’t bring up that Properties panel again… arg… hmm…. think, think. Let’s try running gconf-editor from the command line, for any (former?) Windows users, that’s like the Registry Editor. I try Edit -> Find…, type in openembedded, and check the boxes to search both key names and key values. That gives me the setting, where I change /home/hans/code/openembedded to openembedded, and it works! Phew…

Ok, so you might start to notice a little trend here with these posts. In July, I started work on a big project to create easy-to-use firmwares for old embedded devices called Reware (thanks to Turtle Kalus for the name). So I’ve been hacking my own Palm TX, and of course, discovering annoying, technical details in the process.

So now for my latest issue: So now I’ve been booting my Palm TX using HackNDev’s GNU/Linux images off of an SD card. It works really well, thanks to things like Cocoboot, the little Palm app that starts booting the Linux kernel. But now I’ve noticed that whenever I reboot back into PalmOS from GNU/Linux, my wifi behaves badly. Basically, when I try to connect to a wifi network, it doesn’t use the network I select. Instead, it uses the name that is on top of the list of the previous networks that aren’t currently enabled. It’s maddening, no matter which active network I try, it always selects the topmost inactive network.

Luckily, I found a simple solution! Eureka! I merely turn on the wifi, it doesn’t need to be connected, just on. Then I hit the reset button on the back of my Palm TX. Voila! The wifi works properly again.

Just for fun, here’s my Palm TX booting HackNDev’s image:

So I just spent a joyous few hours trying to figure out how to format my 2GB SD card for my Palm TX and have it work both on my laptop and on the Palm. The Palm will happily format the card itself, and it’ll work just fine inside the Palm, but alas, nowhere else. So then I tried formatting the card on my laptop, both with Ubuntu and with Mac OS X, and then the Palm just wants to format it when I stick it in.

So I dove in deep into the various geeky tools to see what I could do. I tried to repartition the card using cfdisk /dev/sdb1 but this lovely Palm format creates something that cfdisk doesn’t recognize:

FATAL ERROR: Bad primary partition 0: Partition ends in the final partial cylinder

Using fdisk allowed me to see that the partition and the filesystem were marked as FAT16, so I could mount it if I told mount which filesystem it should see, like this: mount -t vfat /dev/sdb1 /mnt/tmp. So I can mount it ok, but not everything is happy:

[93139.773051] attempt to access beyond end of device
[93139.773067] sdb: rw=0, want=2797588, limit=2011136

Hmm, something is amiss. My guess is that this SD card is slightly bigger than 2GB in actual capacity, and FAT16 has a hard 2GB limit. The Palm is joyfully formatting the whole thing as FAT16, even though it should stop at 2GB. Anyway, back to making things work. I read lots of forums, tried some new tools like mkdosfs -F 16 /dev/sdb1 and dosfsck /dev/sdb1, but still it wasn’t working. I had read somewhere that Palms didn’t support FAT32, which is the FAT for disks bigger than 2GB, so I didn’t try that… until now.

So I created a new partition using the lovely cfdisk and set the partition type to 0B W95 FAT32, then I used mkdosfs /dev/sdb1 to create a FAT32 filesystem (mkdosfs tries to use the most appropriate version of FAT). And…. voila! It works on my Palm TX, it works on Ubuntu, it works on Mac OS X. And I am happy…