Supporting self-signed certificates

One lovely thing about generating your own HTTPS self-signed certificates is that you can set the expiration date to whatever you want it to be.  With Certificate Authority companies, they don’t give that option. Most force you to renew every year, some might give you the option of renewing every two years if you want to pay even more.  When you generate a self-signed certificate, the expiration date is just a command line option that nothing else is going to second guess.  For example, this would be valid for about 27 years:

openssl x509 -req -days 10000 -in server.csr -signkey server.key -out server.crt

The problem is of course you get a big fat certificate warning in every browser, and some software just won’t let you connect at all. But all is not lost! On Linux-based systems like Debian GNU/Linux, Ubuntu, or Android, you can just install your self-signed certificate just like all of the official certificates. The best way to do this is start with the certificate .pem or .crt file that was generated on the webserver itself, and copy that around using a USB thumb drive. Then you can import it into Firefox in the preferences: Advanced -> Certificates -> View Certificates, then the Servers and Authorities tab, then Import…

Then lots of other things rely on the system-provided certificates. You can add yours there too:

sudo cp /usr/share/ca-certificates/
sudo chown root.root /usr/share/ca-certificates/
sudo chmod 0755 /usr/share/ca-certificates/
sudo update-ca-certificates --fresh

Google Chrome and Chromium do not use the system certificates, nor do they provide an interface to do this on GNU/Linux, so you have to use a command line tool:

certutil -d sql:$HOME/.pki/nssdb -A -t "P,," \
   -n -i /usr/share/ca-certificates/

On Android, you can also install this file, but unless you have root on your device, you will always have this big fat warning. To install on Android:

  1. Copy your certificate file to your SD Card:

    adb push /sdcard/

  2. Go to Settings -> Security -> Install from storage
  3. choose your certificate file

Then you can either manually copy your cert file into the right place as root, or use an app that does it for you. I used Move Certs! install from F-Droid. Then go to Settings -> Security -> Clear Credentials to get rid of the warning, since your certificate is now listed under SYSTEM now anyhow.

Parsing Baidu Map URLs

Latitude and longitude (latlong) are a very old standard way of expressing a location on this planet.  But not everyone likes to follow standards in software, lots of services want to try to force users to use their services, so they hide simple data like latitude and longitude behind their own custom schemes.  But latlong seems very hard for the map people to avoid, and rightly so.  So that means you can usually parse out the latlong from a URL for an online map.  The last piece of data that is quite useful on online maps is the “zoom”, i.e. how close the view is.  That is usually parsible as well.

Lots of services use shorturls to make it easy to share a long and complicated URL. OpenStreetMap has a good one:  It uses a open algorithm to convert latitude, longitude, and zoom to a short base64-ish string.  That means that apps can get the datapoint from an OpenStreetMap URL without having to query some online server, or even have internet access at all.

But sadly, most shortlinks are used by companies as a tracking service, and they keep their algorithm proprietary.  They want to force you go to their servers so they can track you.  For example, or  Neither of these are parseable for lat/long/zoom as far as I have found.  But they do both lead ultimately to URLs that are parsable.  There are lots of resources for parsing Google URLs, so I’ll leave that to you to find.  But I didn’t find anything on how to parse Baidu URLs, so here’s what I found:,4889173&s=gibberish

  • The s=gibberish query string does not seem to affect the location, but it needs to be present for the link to work.
  • c=13748138,4889173 looks suspiciously like a latlong, but the two numbers are integers
  • l=13 looks like zoom

Turns out that c is a integer presentation of latlong.  I found out that by manually trying numbers in and seeing where on the map it displayed.  c=0,0 is off the coast of West Africa, like it should be.  And c=18000000,0 is halfway around the world.  So the values are between 0 and 18,000,000. Entering values into l shows its clearly zoom.  So voila!  There we have it.

privacy leaks when using shortlinks for location

I’m digging through the guts of how apps are sharing location information. For the most part, its not a great situation, but there is a lot of hope with the geo: URI ( Anyway, lots of Google apps share location using a shortlink (e.g., which is even worse for a number of reasons:

  1. it uses http: to connect to
  2. the link that the redirects to is also http://, so even if you do, then the next step is http://.
  3. the link that redirects do obfuscates the latlong, so it can’t be parsed out of it by other apps, even though the final link when you get to the page includes the latlong (very lame, Google).

Here’s an example flow (note the middle step has HTTP):

So I’m looking for ideas on how an app can get the latlong securely. One simple way to improve this situation would be to pass something in the query string to to make it only use HTTPS, but Google does not seem to provide such a thing.  It would also be helpful to be able to parse the ftid thing, since it requires at least a 10k download to get the latlong from the final page.

Otherwise, I think an app will have to actually connect to, then get the redirect URL and convert it to HTTPS.

Rooting a $20 tablet: Azpen A727

I just picked up a little 7″ tablet (really more like 6″) for $20 on a special. It normally sells for $50, which is still really damn cheap. It is a totally workable Android 4.2 tablet, definitely worth $50 if you want a tablet, or a little extra thing to kick around.

But being me, I of course must have root on every device I own. Turns out it was dead simple, really, it came rooted. Here’s how I installed Superuser on it so I can use apps that use root.

  1. first download this:
  2. unzip
  3. adb root (to run adb as root user on the emulator)
  4. adb remount
  5. adb push Superuser.apk /system/app/
  6. adb push armeabi/su /system/xbin/su
  7. adb shell chmod 6755 /system/xbin/su
  8. adb shell ln -s /system/xbin/su /system/bin/su
  9. wget
  10. adb push init.superuser.rc /

And for those who want to live on the bleeding edge, try out the new system app mode of the F-Droid app store, which makes the install process smoother:

Then go into the FDroid preferences, and check off “Install using system permissions”.

installing koush’s Superuser on an Android emulator

I want to install Superuser on an Android emulator so that I can have automated testing of our apps that use root access. So far, it has been a bit tricky, but I finally got it working.

Here’s how:

  1. first download this:
  2. unzip
  3. adb root (to run adb as root user on the emulator)
  4. adb remount
  5. adb push Superuser.apk /system/app/
  6. adb push armeabi/su /system/xbin/su
  7. adb shell chmod 6755 /system/xbin/su
  8. adb shell ln -s /system/xbin/su /system/bin/su
  9. adb shell "su --daemon &"
  10. adb shell rm /system/app/SdkSetup.apk

LilDebi seems to be running the install fine and created /data/debian, so it is working. I don’t know if this setup will survive a reboot of the emulator though.

simple automated backup to a USB disk on Debian/Ubuntu/Mint

I used to use Mac OS X as my main machine, indeed for 18 years since I started when it was called NeXTSTEP.  Now Apple has turned it into a paradise for mindless consumption, driving creators away.  Now I use Linux Mint as my daily system, but one thing I was missing was a really simple, automatic backup to an external USB disk.  Apple did a good job with Time Machine.  Its simple to setup, happens automatically and in the background, provides a directory structure that you can browse like any standard file system and see the complete snapshots of your computer at the time of a given backup.

So I looked at a lot of apps and tools for doing this on Mint.  I looked at Back In Time, simplebackup, pybackpack, obnam, BoxBackup, rsnapshot, and more.  Back In Time seemed promising, and I probably should have considered it more. Its totally GUI focused, and oriented towards backing up the home folder as a regular user. I want a full disk backup as root. Also, the UI hit me with a ton of options, and I didn’t want to think that much.  simplebackup just seemed like a simple way to make a single backup, with no snapshots. pybackpack is no longer developed.  obnam sounds very promising as a command line tool, but it does not represent the backups as transparent files on a filesystem, so that nixed it for me.  BoxBackup has a client/server architecture, so far too complicated.  I think its oriented towards backing up a fleet of servers to a single backup host.

rsnapshot ended up being quite simple to set up for a command line person like me, and also easy to automate.  And most importantly, it backs up easily to an external disk and writes out each snapshot as a full image of the system at the time of the snapshot.  Sold!  So I just needed to tweak it a little bit.

I had to make a couple edits to /etc/rsnapshot.conf.  One annoying detail to note in this config file is that you have to use tabs between the config items in a line, spaces won’t work. So first I pointed it to a folder on my external USB backup disk:

snapshot_root /media/ext4backup/palatschinken/

Then I told it not to create the root folder on the backup disk. If the backup disk is not present, it should not try to backup at all:

no_create_root 1

Then I enabled one fs mode (i.e. -x in rsync speak), which means that rsnapshot won’t cross the boundaries of a disk. So if you tell it to backup /, and you have /home on a separate partition or disk, it won’t backup /home. You could then add /home specifically, then it would be backed up. Here’s the option:

one_fs 1

Then I specified what I want backed up. These are each separate partitions on my system:

backup / root/
backup /media/share share/

Now the rsnapshot should happily backup. You can test it by running rsnapshot hourly.So the next step is to cron it so it runs automatically in the back ground. I made a script like this for each of the cron folders (/etc/cron.hourly, /etc/cron.daily, /etc/cron.weekly, /etc/cron.monthly):


# See ionice(1)
if [ -x /usr/bin/ionice ] &&
    /usr/bin/ionice -c3 true 2>/dev/null; then
    IONICE="/usr/bin/ionice -c3"

test -d /media/ext4backup/palatschinken && \
	nice $IONICE /usr/bin/rsnapshot hourly

Be sure to change the hourly in each script to match the folder its in. Adding in nice and ionice makes it run at very low CPU and disk priority, so you don’t notice that its running really. Lastly, if you are using this on a machine that is shutodwn or goes to sleep often, then you probably want to install anacron to make sure the daily, weekly, monthy jobs run (apt-get install anacron, it usually configures itself to do the right thing once its installed).

Converting MediaWiki to Redmine

We had a MediaWiki install for our wiki needs, and we used it lots and filled it up with good stuff. Then we got Redmine going. Redmine has its own integrated wiki, but with a different wiki syntax. We had all this lovely scribblings in our mediawiki, but we wanted the tasty redmine integration. So conversion must be done. Thanks to python, it was easy. Here’s the script I wrote:


from BeautifulSoup import BeautifulSoup
import io
import os
import re
import sys
import urllib2

# def download_inline_images(text):
#     m = re.findall()
#     for inline in m:
#         print('Downloading ' + inline)
#         image = urllib2.urlopen('' + inline)
#         with open(inline, 'w') as f:
#             f.write(

def convert_and_save(title, text):
    contents = 'h1. ' + re.sub('_', ' ', title) + '\n\n'
    contents += text

    # headers
    contents = re.sub('===== (.+?) =====', 'h5. \\1\n', contents)
    contents = re.sub('==== (.+?) ====', 'h4. \\1\n', contents)
    contents = re.sub('=== (.+?) ===', 'h3. \\1\n', contents)
    contents = re.sub('== (.+?) ==', 'h2. \\1\n', contents)
    contents = re.sub('= (.+?) =', 'h1. \\1\n', contents)

    # fix bullet lists
    contents = re.sub('\n\*\*([^ ])', '\n** \\1', contents)
    contents = re.sub('\n\*([^ \*])', '\n* \\1', contents)

    # bold - needs to be after 'fix bullet lists'
    contents = re.sub("'''(.+?)'''", '*\\1*', contents)

    # italic
    contents = re.sub("''(.+?)''", '_\\1_', contents)

    # external links with mistaken double brackets
    contents = re.sub('\[\[(http.+?) (.+?)\]\]', '"\\2":\\1', contents)

    # external links
    contents = re.sub('\[(http.+?) (.+?)\]', '"\\2":\\1', contents)

    # inline image links
    contents = re.sub('\[\[File:([^|\]]+).*?\]\]', '!\\1!', contents)

    # make sure there's a trailing newline
    contents += '\n'

    filename = re.sub('/', '_', title) + '.redmine'
    with, 'w', encoding='utf8') as f:

def crawl_url(url):
    page = urllib2.urlopen(url)
    soup = BeautifulSoup(page)
    find = soup.find('textarea')
    if not find or len(find) == 0:
    textarea = find.contents[0]

    m = re.match('.*title=([^&]+)', url)
    title =
    convert_and_save(title, textarea)

    m = re.findall('\[\[([^|\]]+)', textarea)
    for page in m:
        print 'Trying: ' + page
        if not os.path.exists(page + '.redmine') \
                and not page.startswith('File:') \
                and not page.startswith('http'):
            print 'MATCH: ' + page
            crawl_url('' + re.sub(' ', '_', page) + '&action=edit')

def main(argv):
    if len(argv) == 0:
        print "Usage: " + sys.argv[0] + " [URL]"
    if len(argv) == 1:

if __name__ == "__main__":


Get every new post delivered to your Inbox.