Skip to content
GRENDELMAN.NET
  • Home
  • Photos
  • Projects & Pages
    • Projects
    • Files
    • Humor
    • Bookmarks
  • Login

We zijn vertrokken!

Posted on: Saturday 7 November 2015 /
Categories: Nieuw-Zeeland

Dat wil zeggen: we hebben Nederland alweer bijna een week achter ons gelaten, maar vanochtend zijn we dan begonnen aan onze camper-rondreis.

De aanloop naar de reis, nog in Nederland, was niet helemaal zonder obstakels: zowel Marijke als onze kat Bufke hebben op ‘t laatste moment nog een flinke tandarts-behandeling ondergaan, Rowan had de nacht voor vertrek diarree en heeft overgegeven en de trein naar Schiphol die we wilden nemen reed niet. Maarr… zondagochtend om 07.49u stipt hebben we Eindhoven dan toch achter ons gelaten.

Op Schiphol zijn we uitgezwaaid door onze ouders, en precies op tijd vertrokken we met het vliegtuig naar Hong Kong. De vlucht verliep redelijk. Rowan kon liggen en heeft flink wat uurtjes geslapen, maar helaas gold dat niet voor ons. In Hong Kong hebben we snel ingecheckt in ons daghotel en hebben we lekker gedoucht en nog 4 uurtjes geslapen.

Om ‘n uur of 2 ‘s middags hebben we de Airport Express naar Hong Kong centrum genomen, en hebben we een stuk langs het water gelopen, naar het Hong Kong Observation Wheel, een flink reuzenrad met mooi uitzicht over de stad. Na nog een broodje en een drankje hadden we rond half 6 weer de trein naar het vliegveld.

20151101072443.jpg 
Klaar voor vertrek!

2015-11-01 07:24:43
© 2015 Martijn Grendelman
20151101074826_IMG_8546.JPG 
In de trein naar Schiphol

2015-11-01 07:48:26
© 2015 Martijn Grendelman
20151101074859_IMG_8548.JPG 
In de trein naar Schiphol

2015-11-01 07:48:59
© 2015 Martijn Grendelman
20151101075145_IMG_8549.JPG 
In de trein naar Schiphol

2015-11-01 07:51:45
© 2015 Martijn Grendelman
20151101075213_IMG_8551.JPG 
In de trein naar Schiphol

2015-11-01 07:52:13
© 2015 Martijn Grendelman
20151101113502_IMG_8554.JPG 
Cathay Pacific vlucht CX270

2015-11-01 11:35:02
© 2015 Martijn Grendelman
20151101124106.jpg 
Ready for departure

2015-11-01 12:41:06
© 2015 Martijn Grendelman
20151101220827.jpg
Slapen in het vliegtuig lukt... een beetje

2015-11-01 22:08:27
© 2015 Martijn Grendelman
20151101220854.jpg
Slapen in het vliegtuig lukt... een beetje

2015-11-01 22:08:53
© 2015 Martijn Grendelman
20151101234047.jpg
We zijn d'r bijna....

2015-11-01 23:40:46
© 2015 Martijn Grendelman
20151102132529_IMG_8561.JPG 
Kamer in het Regal Airport Hotel

2015-11-02 13:25:29
© 2015 Martijn Grendelman
20151102141628.jpg 
Het gezonde menu van de mini-bar

2015-11-02 14:16:28
© 2015 Martijn Grendelman
20151102143226.jpg 
In de hal van het hotel

2015-11-02 14:32:26
© 2015 Marijke Olzheim
20151102143656.jpg 
In de hal van het hotel

2015-11-02 14:36:56
© 2015 Martijn Grendelman
20151102145903.jpg
In de trein naar Hong Kong

2015-11-02 14:59:02
© 2015 Martijn Grendelman
20151102150145.jpg
In de trein naar Hong Kong

2015-11-02 15:01:45
© 2015 Martijn Grendelman
20151102153409_IMG_8565.JPG 

2015-11-02 15:34:09
© 2015 Martijn Grendelman
20151102153458_IMG_8566.JPG 
Hong Kong Observation Wheel

2015-11-02 15:34:58
© 2015 Martijn Grendelman
20151102153536_IMG_8567.JPG 

2015-11-02 15:35:36
© 2015 Martijn Grendelman
20151102153614_IMG_8568.JPG 

2015-11-02 15:36:14
© 2015 Martijn Grendelman
20151102155002_IMG_8571.JPG 

2015-11-02 15:50:02
© 2015 Martijn Grendelman
20151102155913_IMG_8572.JPG 
Uitzicht over Victoria Harbour

2015-11-02 15:59:13
© 2015 Martijn Grendelman
20151102155942_IMG_8574.JPG 
Rowan in het Observation Wheel

2015-11-02 15:59:42
© 2015 Martijn Grendelman
20151102160008_IMG_8575.JPG 
Tsim Sha Tsui

2015-11-02 16:00:08
© 2015 Martijn Grendelman
20151102160015_IMG_8576.JPG 
Uitzicht over Victoria Harbour

2015-11-02 16:00:15
© 2015 Martijn Grendelman
20151102160028_IMG_8577.JPG 
Uitzicht over Victoria Harbour

2015-11-02 16:00:28
© 2015 Martijn Grendelman
20151102160040_IMG_8578.JPG 
Marijke in het Observation Wheel

2015-11-02 16:00:40
© 2015 Martijn Grendelman
20151102160506_IMG_8580.JPG 

2015-11-02 16:05:06
© 2015 Martijn Grendelman
20151102160523_IMG_8581.JPG 

2015-11-02 16:05:23
© 2015 Martijn Grendelman
20151102160609_IMG_8583.JPG 

2015-11-02 16:06:09
© 2015 Martijn Grendelman
20151102161451_IMG_8584.JPG 

2015-11-02 16:14:51
© 2015 Martijn Grendelman
20151102162937_IMG_8585.JPG 

2015-11-02 16:29:37
© 2015 Martijn Grendelman
20151102164737.jpg 

2015-11-02 16:47:37
© 2015 Marijke Olzheim
20151102170301.jpg 

2015-11-02 17:03:01
© 2015 Martijn Grendelman
20151102175531.jpg 

2015-11-02 17:55:30
© 2015 Martijn Grendelman
20151102184105.jpg

2015-11-02 18:41:05
© 2015 Martijn Grendelman
20151102192939.jpg 

2015-11-02 19:29:39
© 2015 Martijn Grendelman
20151102192950.jpg 

2015-11-02 19:29:50
© 2015 Martijn Grendelman
20151102201129_IMG_8587.JPG 

2015-11-02 20:11:29
© 2015 Martijn Grendelman
20151102215521.jpg

2015-11-02 21:55:20
© 2015 Martijn Grendelman

‘s Avonds om 21:15u (met een kwartiertje vertraging) vertrok onze vlucht naar Auckland. We hadden in het vliegtuig een rij vooraan gekozen, zodat we Rowan op de grond konden laten slapen. Al snel hadden we een bedje voor hem gemaakt van een paar dekens en hij sliep echt binnen een paar minuten. Helaas kwam er al snel een steward vertellen dat de safety regulations het niet toestonden om op de grond te slapen. Maar op de stoelen slapen ging ook niet, want de stoelen op deze rij hadden geen opklapbare armleuningen. Gelukkig was er een paar rijen achter ons iemand bereid om van plaats te ruilen, en we hebben Rowan al slapend verplaatst. Helaas hebben wij zelf ook deze vlucht nauwelijks geslapen, dus we waren enigzins vermoeid toen we op dinsdagmiddag in Auckland aankwamen.

In Auckland duurt het wel even voordat je het land binnen bent. Nieuw-Zeeland heeft strenge regels als het gaat om het importeren van natuurproducten en je moet alles (van een appel tot de modder onder je wandelschoenen) aangeven bij de biosecurity screening. En als er tegelijk met jou net drie A380’s zijn geland, dan duurt dat even. Uiteindelijk viel dat toch wel mee, en rond 14:15u plaatselijke tijd troffen we John, onze AirBNB host die ons met de auto kwam ophalen, in de aankomsthal.

Bij John en Emma thuis hebben we onze bagage uitgepakt, heeft Marijke wat boodschappen gedaan en hebben we snel wat gegeten. Om 18u vielen we alledrie om van de slaap, en toen Rowan in bed lag zijn wij ook maar gaan slapen. Uiteraard werden we midden in de nacht wakker, en ik denk dat alleen Rowan na 4 uur ‘s nachts nog geslapen heeft. Overigens zijn we nu 2 dagen verder, maar we hebben allemaal nog steeds flink last van de jetlag. Hopelijk gaat ‘t vannacht beter.

Woensdag hebben we het rustigaan gedaan. Martijn wilde graag Auckland in om een glimp op te vangen van de huldiging van de All Blacks, het nationale rugbyteam, die ‘s middags in de stad werd gehouden, maar we hebben toch maar besloten om het bij een wandelingetje in de buurt te houden. John en Emma wonen vlak bij bos en kust, dus dat was best de moeit waard. Onze gastheer en gastvrouw zijn beide Engels van origine, maar wonen al flink wat jaartjes in Nieuw-Zeeland. Ze verhuren de onderverdieping van hun huis via AirBNB en hun gastvrijheid kent bijna geen grenzen. We hebben de hele avond bij hen aan de keukentafel zitten kletsen, onder het genot van een paar glaasjes wijn.

20151103151031_IMG_8589.JPG 

2015-11-03 15:10:31
© 2015 Martijn Grendelman
20151104095415_IMG_8590.JPG 

2015-11-04 09:54:15
© 2015 Martijn Grendelman
20151104122624_IMG_8592.JPG 

2015-11-04 12:26:24
© 2015 Martijn Grendelman
20151104122725.jpg 

2015-11-04 12:27:25
© 2015 Marijke Olzheim
20151104123650_IMG_8593.JPG 

2015-11-04 12:36:50
© 2015 Martijn Grendelman
20151104123857_IMG_8595.JPG 

2015-11-04 12:38:57
© 2015 Martijn Grendelman
20151104123938.jpg 

2015-11-04 12:39:37
© 2015 Marijke Olzheim
20151104124042_IMG_8597.JPG 

2015-11-04 12:40:42
© 2015 Martijn Grendelman
20151104124342.jpg 

2015-11-04 12:43:42
© 2015 Marijke Olzheim
20151104124418_IMG_8601.JPG 

2015-11-04 12:44:18
© 2015 Martijn Grendelman
20151104125450_IMG_8603.JPG 

2015-11-04 12:54:50
© 2015 Martijn Grendelman
20151104125942.jpg 

2015-11-04 12:59:41
© 2015 Marijke Olzheim
20151104125952_IMG_8604.JPG 

2015-11-04 12:59:52
© 2015 Martijn Grendelman
20151104131723_IMG_8605.JPG 

2015-11-04 13:17:23
© 2015 Martijn Grendelman
20151104132923_IMG_8606.JPG 

2015-11-04 13:29:23
© 2015 Martijn Grendelman
20151104133031_IMG_8607.JPG 

2015-11-04 13:30:31
© 2015 Martijn Grendelman
20151104150851.jpg 

2015-11-04 15:08:51
© 2015 Marijke Olzheim
20151105082955_IMG_8609.JPG 

2015-11-05 08:29:55
© 2015 Martijn Grendelman
20151105094149.jpg 

2015-11-05 09:41:48
© 2015 Marijke Olzheim

Donderdag was het dan tijd om onze camper op te gaan halen. Bij gebrek aan fatsoenlijk openbaar vervoer in Auckland hebben we een taxi genomen naar de afhaallokatie, vlakbij het vliegveld. Na het bekijken van een filmpje over wat praktische zaken konden we de camper in ontvangst nemen. De fietsen die we erbij gehuurd hadden, moesten we overigens ergens anders ophalen, dus dat was de volgende activiteit. Gelukkig konden we bij onze mountainbikes ook een kinderzitje voor Rowan krijgen, dus we kunnen deze vakantie met z’n drieen met de fiets op stap. Toch wilden we voor Rowan eigenlijk ook nog een eigen fietsje, en de verhuurder wist na een telefoontje een betaalbaar kinderfietsje te staan bij een Bike Barn fietsenwinkel, dus dat werd onze volgende bestemming. Met onze mega-camper, links rijdend door de drukte van het stadsverkeer werden we gelijk in het diepe gegooid. Uiteindelijk waren we rond 16u pas weer terug bij John en Emma. ‘s Avonds hebben we wat spullen ingepakt en heeft John ons wat reistips aan de hand gedaan.

Vrijdag waren we allemaal vroeg wakker en na een hartelijk afscheid van John en Emma waren we
rond kwart over 10 op weg. Op ons gemak zijn we gaan rijden in de richting van Paihia, bij de Bay of Islands in het noorden van het land. Na diverse stops onderweg (onder andere voor boodschappen) kwamen we rond 16 uur aan op het Beachside Holiday Park, waar we een mooi plekje tussen het water en de speeltuin hebben gevonden. Hier hebben we onze camper een beetje beter ingericht en na het eten snel Rowan naar bed gebracht. En nu is het bijna 22u op vrijdagavond en zit ik (Martijn) met licht-verkleumde vingertjes dit verhaal te typen 🙂

(En door netwerk-problemen is het inmiddels zaterdagavond voor ik dit verhaal kan posten, maar onze avonturen van vandaag bewaar ik voor de volgende keer).

20151105111922_IMG_8610.JPG 

2015-11-05 11:19:22
© 2015 Martijn Grendelman
20151105133612_IMG_8613.JPG 

2015-11-05 13:36:12
© 2015 Martijn Grendelman
20151105133753.jpg 

2015-11-05 13:37:53
© 2015 Marijke Olzheim
20151105163923_IMG_8615.JPG 

2015-11-05 16:39:23
© 2015 Martijn Grendelman
20151105174014.jpg 

2015-11-05 17:40:14
© 2015 Martijn Grendelman
20151105212943_IMG_8616.JPG 

2015-11-05 21:29:43
© 2015 Martijn Grendelman
20151105213416_IMG_8624.JPG 

2015-11-05 21:34:16
© 2015 Martijn Grendelman
20151105213441_IMG_8625.JPG 

2015-11-05 21:34:41
© 2015 Martijn Grendelman
20151105213628_IMG_8626.JPG 

2015-11-05 21:36:28
© 2015 Martijn Grendelman
20151106093036.jpg 

2015-11-06 09:30:36
© 2015 Martijn Grendelman
20151106111555_IMG_8629.JPG 

2015-11-06 11:15:55
© 2015 Martijn Grendelman
20151106120209.jpg 

2015-11-06 12:02:09
© 2015 Martijn Grendelman
20151106125258.jpg 

2015-11-06 12:52:57
© 2015 Marijke Olzheim
20151106140805.jpg 

2015-11-06 14:08:04
© 2015 Martijn Grendelman
20151106141747.jpg 

2015-11-06 14:17:46
© 2015 Marijke Olzheim
20151106155955.jpg 

2015-11-06 15:59:55
© 2015 Marijke Olzheim
20151106160111.jpg 

2015-11-06 16:01:11
© 2015 Martijn Grendelman
20151106161416_IMG_8638.JPG 

2015-11-06 16:14:16
© 2015 Martijn Grendelman
20151106161525_IMG_8641.JPG 

2015-11-06 16:15:25
© 2015 Martijn Grendelman
20151106161628.jpg 

2015-11-06 16:16:28
© 2015 Martijn Grendelman
20151106162424.jpg 

2015-11-06 16:24:23
© 2015 Marijke Olzheim
20151106162502.jpg 

2015-11-06 16:25:02
© 2015 Martijn Grendelman
20151106163001_IMG_8642.JPG 

2015-11-06 16:30:01
© 2015 Martijn Grendelman
20151106175214_IMG_8646.JPG 

2015-11-06 17:52:14
© 2015 Martijn Grendelman
20151106193019.jpg 

2015-11-06 19:30:19
© 2015 Martijn Grendelman
20151107092910_IMG_8647.JPG 

2015-11-07 09:29:10
© 2015 Martijn Grendelman
20151107092955_IMG_8648.JPG 

2015-11-07 09:29:55
© 2015 Martijn Grendelman
20151107093006_IMG_8649.JPG 

2015-11-07 09:30:06
© 2015 Martijn Grendelman
20151107093030_IMG_8650.JPG 

2015-11-07 09:30:30
© 2015 Martijn Grendelman
20151107094316_IMG_8652.JPG 

2015-11-07 09:43:16
© 2015 Martijn Grendelman

Volgende: Kawakawa en Orewa

Fast & frequent incremental ZFS backups with zrep

Posted on: Friday 8 May 2015 /
Categories: Nerd Stuff

Recently, I replaced one of the Windows fileservers at $dayjob with a Debian-based Samba server. For the data storage, I chose ZFS on Linux. I have been running ZFS on Linux on my backup servers for a while now, and it seems that with the latest release (0.6.4.1, dated April 23 2015), most, if not all of the stability problems I had with earlier versions are gone.

ZFS backups

ZFS has a few features that make it really easy to back up efficiently and fast:

  1. Cheap snapshots.
  2. ZFS send / receive.

ZFS snapshots are cheap in that they don’t cost any performance. ZFS is a copy-on-write filesystem, which means that every block that needs changing, is first copied to a new block, and only after the update succeeds, the reference in the filesystem is updated to the new block. The old block is then freed, unless it is part of a snapshot, in which case the data is simply left intact. You can make many snapshots of a single ZFS filesystem (theoretically 264 in a storage pool) without any cost other than the disk space they consume, which is equal to the growth of your data (or rather: accumulated size of all changes) since the oldest snapshot.

ZFS allows you to take a shapshot and send it to another location as a byte stream with the zfs send command. The byte stream is sent to standard output, so you can do with it what you like: redirect it to a file, or pipe it through another process, for example ssh. On the other side of the pipe, the zfs receive command can take the byte stream and rebuild the ZFS snapshot. zfs send can also send incremental changes. If you have multiple snapshots, you can specify two snapshots and zfs send can send all snapshots inbetween as a single byte stream.

So basically, creating a fast incremental backup of a ZFS filesystem consists of the following steps:

  1. Create a new snapshot of the filesystem.
  2. Determine the last snapshot that was sent to the backup server.
  3. Send all snapshots, from the snapshot found in step 2 up to the new snapshot created in step 1, to the backup server, using SSH:
zfs send -I <old snapshot> <new snapshot> | ssh <backupserver> zfs receive <filesystem>

Of course, on the backup server you can leverage some of the other great features of ZFS: compression and deduplication.

Enter zrep.

Zrep

Zrep is a shell script (written in Ksh) that was originally designed as a solution for asynchronous (but continuous) replication of file systems for the purpose of high availability (using a push mechanism). It was later expanded with the possibility to create backups of a filesystem using a pull mechanism, meaning the replication is initiated from the backup server and no SSH access is needed to the backup server, as it would be with push-replication.

Zrep is quite simple to use and it has good documentation, although setting it up for use as a backup solution took me a few attempts to get right. I won’t go into the gory details here, but I’ll describe my setup.

It basically works like this:

  • Zrep needs to be installed on both sides.
  • The root user on the backup server needs to be able to ssh to the fileserver as root. This has security implications, see below.
  • A cron job on the backup server periodically calls zrep refresh. Currently, I run two backups hourly during office hours and another two during the night.
  • Zrep sets up an SSH connection to the file server and, after some sanity checking and proper locking, calls zfs send on the file server, piping the output through zfs receive:
ssh <fileserver> zfs send -I <old snapshot> <new snapshot> | zfs receive <filesystem>
  • Snapshots on the fileserver need not be kept for a long time, so we remove all but the last few snapshot in an hourly cron job (see below).
  • Snapshots on the backup server are expired and removed according to a certain retention schedule (see below).

SSH access and security

Since all ZFS operations, like making snapshots and using send / receive require root privileges (at least on Linux by default; other OSs like Solaris are more flexible in this, and even Linux may allow you to chown/chmod /dev/zfs to delegate these privileges – see this issue on Github for more information) , zrep must also run as root on both ends. This means that root needs SSH access to the fileserver, which could be a huge security problem. What I usually do to mitigate this as much as possible, is:

  1. SSH access is firewalled and only allowed from IPs that need to have access. Between datacenters, I use VPNs and externally routeable IP adresses generally do not have SSH access.
  2. PasswordAuthentication no
  3. PermitRootLogin forced-commands-only
  4. Use an SSH keypair specific to this application, and configure an entry for the fileserver in /root/.ssh/config on the backup server, using this key.
  5. Use a wrapper script for zrep and specify this as a forced command in root’s authorized_keys.
  6. Use a list of ‘from’ IPs (containing only your backup server(s)) for this specific key in root’s authorized_keys to restrict access even beyond the firewall.

The wrapper script can check the command it gets from the client and only exec the original command if nothing smells fishy.

I guess it would also be possible to use sudo, instead of granting root SSH access, but I haven’t tested this. If you would like to try it out: zrep allows for specifying the path to the zrep executable on the remote end using an environment variable (ZREP_PATH), so maybe it’s as easy as calling:

ZREP_PATH="sudo zrep" zrep refresh <pool/fs>

Since zrep on the backup server would still run as root, you would need to configure the user account to use in the SSH connection in /root/.ssh/config. For example:

Host myfileserver
    User zrep
    IdentityFile /root/.ssh/id_rsa_zrep

And of course you would need to configure sudo on the fileserver to allow the user of choice to execute zrep (or the wrapper script) as the root user without specifying a password. Let me know if it works, when you give it a try.

Backup retention / expiration

When using the ‘refresh‘ command, zrep does not automatically expire old snapshots like it does when using the more standard ‘sync’ replication features, so we have to trigger snapshot expiration by hand. Zrep keeps track of the snapshots that it makes and ships to the backup server by recording the timestamps in ZFS custom properties. It also provides an expire command, that lets you clean up old snapshots, but it is very rudimentary. It only allows you to specify a number of snapshots to keep (5 by default, but this value is changeable for local and remote snapshots independently) and if you call zrep expire, it just deletes all but the last 5 snapshots. This is fine for the fileserver itself, so we run an hourly cronjob, just telling zrep to expire all but the last 5 snapshots. The -L flag tells zrep to leave any remote (replicated) snapshots alone:

17 *    * * *   root    /usr/local/bin/zrep expire -L >/dev/null

On the backup server however, I would like a more sophisticated rentention schedule. We make snapshots frequently and even though there is no real technical benefit to cleaning them up, I don’t really want to keep all of them around. I’d like to retain backups to a more traditional schedule, like:

  • every snapshot for the past 48 hours
  • one backup daily for 15 days
  • one weekly backup for 4 weeks
  • one backup monthly for 6 months
  • a yearly snapshot for a couple of years

For this, we use a Python script called zrep-expire (Github). Zrep-expire looks at the creation time of a snapshot and checks it against an expiration rule. If, according to the rule, the snapshot is expired, it destroys the snapshot. The crontab entry looks like this:

55 6 * * * /usr/local/bin/zrep-expire -c /etc/zfs/zrep-expire.conf

Listing and restoring backups

A list of all snapshots with the most interesting properties can be viewed with:

/sbin/zfs list -t snapshot -o name,zrep:sent,creation,refer,used

ZFS on Linux, like Oracle Solaris ZFS, exposes snapshots in a .zfs/snapshot directory in the root of the filesystem. Please note that the .zfs directory is hidden and you will not see it even with ls -a. Each snapshot has an entry with its name in .zfs/snapshot, and the root user can cd into those subdirectories and copy files from there to their original location, or anywhere s/he wants. Remember that snapshots are read-only, so you cannot change any data there.

 

Thanks to Philip Brown, the author of Zrep, for some useful feedback on this post.

Introducing Trackserver WordPress plugin

Posted on: Wednesday 11 February 2015 /
Categories: GPS and mapping, Nerd Stuff

A while ago, I wrote a post titled “Self-hosted live online GPS tracking with Android“, in which I evaluated a few Android apps that can be used for, you guessed it, live online location tracking. The idea is, that you have your phone periodicially send your location to a web service while you’re on the go. The web service stores the location updates and optionally publishes them on a map.

While there are some apps that do this, there aren’t many (free) options for hosting your own server. That’s why I decided to write my own, and do it as a free and open source WordPress plugin, to make it as accessible as possible. Please meet Trackserver.

Trackserver is designed to work with TrackMe and OruxMaps. OsmAnd support is on the roadmap and I’m thinking about adding GpsGate support. All of these apps offer the possibility to periodically post location updates to a web URL that you can specify. Your own WordPress blog with the Trackserver plugin can ‘listen’ on such a URL to receive the updates and store them in a database. The plugin also offers a WordPress shortcode to include maps with your GPS tracks in your posts and pages, using the fantastic Leaflet library. Collecting and publishing your tracks has never been easier, at least not in a self-hosted website!

A short list of Trackserver’s features:

  • Collect location updates using several mobile apps
  • Publish your collected tracks on a map in your posts and pages
  • Upload tracks via an upload form in the WordPress admin backend
  • Upload tracks by sharing a GPX file from your mobile, using a sharing app like Autoshare
  • View, edit, delete and merge your tracks in the WordPress admin backend
  • Live tracking of your location, with automatic updates of the map view in your WordPress post
  • Full-screen map views

Trackserver’s WordPress admin allows you to upload GPX files to add new tracks, view, delete and merge tracks and edit their properties (name, comment, etc.). Please take a look at the plugin page at wordpress.org or the source code repository at Github. Both pages have more information on Trackserver’s features. The current version is 1.0, which has all the basic features described above.

Getting started with Trackserver is quite easy:

  1. Install the plugin in your WordPress site
  2. Install TrackeMe and/or OruxMaps on your Android phone
  3. Configure the app(s) with your WordPress URL, username and password (HowTo with screenshots is included in the plugin)
  4. Start tracking!

Feel free to give it a go, and if you do, let me know what you think!

 

Tags: android, GPS, oruxmaps, osmand, plugin, tracking, trackme, wordpress

First time at FOSDEM

Posted on: Monday 2 February 2015 /
Categories: Nerd Stuff, Personal

Last Saturday, I went to FOSDEM for the first time and I though I’d write a little post about my general findings.FOSDEM

I attended about 7 talks in 5 different tracks. Contentwise, most talks were interesting enough and most speakers I have seen did a decent job on their presentation. Some weren’t quite the masters of the English language you would hope for, and some really need to improve on their time-management (*), but on average it wasn’t bad.

(*) I attended one talk where the speaker started with an introduction of the product he was talking about, before starting on the actual subject of the talk, but due to taking questions from the audience, he only had 5 minutes left after the introduction and he ended up skipping more than half his slides. Since I was already familiar with the product, I learnt nothing new and the whole talk was sort of a waste of time 🙁

There were some things that really bothered me though:

  • The venue has some really old buildings and classrooms, that are noisy as hell. Squeeking and loudly banging doors, and tables being (un)folded were quite distracting. As a matter of fact, I find it hard to imagine that students actually take classes in an environment like that.
  • With almost all the talks I went to, the room was filled to the last chair, giving the impression that bigger rooms are needed.
  • People can walk in and out of talks whenever they like, causing distractions, mostly due to the noise.
  • Talks are scheduled without breaks inbetween, so in the last 5 minutes of a talk (usually the Q&A section) groups of people come barging into the room for the next talk. They can’t sit anywhere, because the room is full, but people wanting to leave the room after the talk now can’t because there are people standing everywhere.

So, in short, I was pleased with the schedule, but on the practical side, I see a lot of room for improvement. I understand that some things are hard to fix, but I would definitely recommend scheduling time between talks and getting people to guard the doors to make sure a talk is finished and people can get out before letting the next crowd in.

In general, FOSDEM is a nice conference to visit. Good talks, and many renowned people from the Open Source community are there (I even saw RMS handing out flyers outside) and they’re available for questions and discussions, which is really great. No entrance fee or registration is required, which makes it very accessible (but it won’t stop me from complaining 😉 ).

FOSDEM has a lot in common with T-DOSE, which is much smaller, but also a great conference.

 

Self-hosted live online GPS tracking with Android

Posted on: Tuesday 30 December 2014 /
Categories: GPS and mapping, Nerd Stuff

Since long I have been interested in live online location tracking using my Android phone and I have been using a collection of software components that makes this possible for a while. The other day, I was discussing the subject with a friend who is looking for a similar solution and has been using OruxMaps for offline tracking. I was unfamiliar with OruxMaps, but it has a zillion features and it looks great in general. It triggered me to do a short survey of the Android landscape, since it appears that more apps have popped up since I last looked.

The solutions I am interested in should be able to do live online tracking to a self-hosted location, meaning the Android app should offer the possibility to enter a custom hostname or URL. Hosted services, like the once popular but now deceased Google Latitude or InstaMapper are not interesting to me. My location data is mine and no one else’s, and I would like to decide how I consume the data and if/how I share my location. This post is categorized in ‘Nerd Stuff‘ for a reason 🙂

So I am looking for:

  • a Android app,
  • open source, if possible,
  • with support for a self-hosted server,
  • AND a matching a server component that can put the location data in a database,
  • OR an open protocol specification or a protocol that is easy to reverse-engineer, so I can build the server component myself.

 

TrackMe

Reviewed version: 1.11.1 (October 2014)TrackMe

TrackMe is available for Windows Phone as well as Android. I have been using it for quite a while. It is robust and offers some interesting features beyond tracking. It also offers a server component written in PHP, which uses a MySQL database for storage. TrackMe is ad-supported, but ads can be removed for €2.24. It supports offline as well as online tracking.

The strong points of TrackMe are:

  • Online tracking when possible (working internet connection), offline tracking and uploading when possible, manually or automatically. It just works.
  • Location updates are grouped in trips.
  • Extensive configuration options, allowing for trade-off decisions between accuracy and battery use.
  • It can be configured to start tracking as soon as the app starts or even when your phone boots.
  • Server component ready to go, using PHP/MySQL. It has a web front-end that plots tracks on a Google Map.
  • Remote control of the app (start/stop tracking, report current location) via SMS
  • Hackable: the online tracking protocol is simple HTTP GET requests, writing your own server is easy.

The drawbacks:

  • Not open source, ad supported, user must dismiss a pop-up offering ad-removal every time the app starts.
  • The Android app does not have a map view, it only displays an activity log.
  • PHP server code is of bad quality and mostly written in 2008. Latest modifications in 2012. Only MySQL is supported as a backend.
  • PHP web front-end uses Google Maps. I prefer OpenStreetMap.
  • I couldn’t get HTTPS to work in the app (‘SSLPeerUnverifiedException: No peer certificate’, even though the server’s SSL certificate is fine).
  • Android app is pretty ugly.
  • Development of the app is really slow.

The bottom line: TrackMe as an Android app would have everything I wanted in a tracking app, if it were a little prettier, if SSL worked and if it were completely free. A map view within the app would be nice, but I can live without it. The PHP server works and the web interface looks alright, but the code is an unmaintainable pile of crap.

Home page: http://www.luisespinosa.com/trackme_eng.html

Play store: https://play.google.com/store/apps/details?id=LEM.TrackMe

OruxMaps

Reviewed version: 6.0.4 (December 2014)Orux Maps

Like I said, I never heard of OruxMaps before yesterday, which is a real shame, because it is the most versatile mapping app I have seen so far. Version 1.0 was released in 2009, so it’s not exactly a new kid on the block either. OruxMaps is free to use and free of ads, but not open source as far as I can tell. OruxMaps is a map-viewer and track-logger with many features, its tagline is “Travel & sport companion“. Just to name a few features that I find most useful:

  • it supports both online and offline maps (in a variety of formats)
  • it can do offline tracking and export to several file formats
  • it can upload tracks to many different services like EveryTrail and ikiMap (many Spanish services, due to the app’s Spanish origin)
  • it can do live online tracking via Map My Tracks, GpsGate and OkMap
  • it supports routes and waypoints, with photos and videos
  • it supports all sorts of external devices, like GPS receivers, ANT+ devices, bluetooth heart rate monitors and so on

and the list goes on and on. This app clearly gets a lot of love from its developer(s?), and I really like that a lot of attention is given to the looks of the app. The icons, the layout of the screen, the menus, even the logo… the whole user experience is excellent, and it seems like every little detail is customizable.

So, I guess I’ll write another post about OruxMaps later, but right now, I’ll focus on the live online tracking features.

Like I said above, live online tracking is supported via 3 different methods, all of which have the option of a self-hosted server solution. The most important one, from Orux’s point of view, is Map My Tracks. This is a service, focused on tracking outdoor activities. It has an open, documented API and OruxMaps offers a setting to override the ‘live tracking’ URL. This makes it possible to implement your own server. Communication takes place over HTTP POST. Map My Tracks uses ‘activities’ to separate tracks, so I think everything we need is here, even though there seem to be some unresolved bugs.

Another interesting option is Orux Maps’ GpsGate support. GpsGate is a commercial product, but the server is free to use up to 5 users. The server runs on Windows, so it may or may not be for you (it’s not for me ;-)), but the GpsGate Server Protocol v2.0 is documented, so implementing a simple server of your own should be a viable option. However, my impression is that the GpsGate support is not widely used in the OruxMaps community, so I can’t say how well it works.

And finally, there’s the OkMap client. OkMap is a Windows desktop application and apparently (I have not tested this in any way), it can listen on a port to which live location updates can be sent. I haven’t found any reference material regarding the used protocol, but I’m pretty sure it isn’t HTTP. Reverse-engineering the protocol and implementing your own server may be a challenge, and with the options of Map My Tracks and GpsGate being there, I don’t see any reason to try.

To me, it seems that Map My Tracks is the easiest choice here, if you were to create your own protocol implementation. I did do a quick search for any Map My Tracks servers that may already be out there, but I didn’t find any.

Home page: http://www.oruxmaps.com/

Play store: https://play.google.com/store/apps/details?id=com.orux.oruxmaps 

OwnTracks

OwnTracksReviewed version: March 2014

OwnTracks (formerly MQTTitude) is a set of open source tools, among which an Android app and one for iOS. The project was started when (and because) Google Latitude was discontinued. Its main purpose is to keep track of your and your friends’ location by showing their picture or a marker on the map. As such, it does not have a notion of ‘trips’ or something like that and location updates are generally infrequent (> 1 minute intervals). Even though the wiki mentions something called ‘Move mode’, which is supposedly meant for actual movement tracking, this feature is not available in the Android app at the time of writing. OwnTracks is designed to be ‘always on’ and cannot be stopped from within the app. All in all, OwnTracks is not really suitable (or meant) as a travel companion.

Another thing that makes OwnTracks a bit of an outsider (but all the more interesting, one might argue), is the fact that it uses MQTT rather than HTTP for communication. This choice was made because MQTT is a really great fit for messages like location updates and it supposedly has great benefits for your phone’s battery life. To use OwnTracks, you need an MQTT broker like Mosquitto. You could also use a public broker, but that kind of defeats the purpose of choosing a self-hosted solution (privacy). Mosquitto is not difficult to set  up, but you would need a place to run it and a standard shared hosting account probably won’t cut it.

Mosquitto is merely a message broker and doesn’t store anything by itself. For that purpose, OwnTracks has a backend script (m2s) that subscribes to the topics that the app publishes to, and stores the updates in a database.

Home page: http://owntracks.org/

Play store: https://play.google.com/store/apps/details?id=st.alr.mqttitude

OsmAnd

Reviewed version: 1.9.4 (December 2014)OsmAnd

OsmAnd is a different beast, because tracking is not its core competence. OsmAnd is a maps and navigation app, that supports offline maps. The reason to list it here is that it can be used for live online tracking, and since you may already use it for navigation, you might as well use it for tracking.

OsmAnd supports tracking through custom web services. Once the tracking functionality is enabled through the plugin manager, a URL can be set and the location is logged on a configurable interval using a simple HTTP GET. HTTPS does not seem to work; my server did not receive any updates when using a HTTPS URL, but no errors were reported in the app. Once the tracking settings are in order, you can start tracking (both offline and online) with a single tap on the screen. OsmAnd using a custom web URL does not use the concept of trips or something like that, it just sends location updates without any context.

Some notes:

  • Tracking settings can be set in 3 different profiles, depending on the type of transport (car / bike / foot).
  • Local (offline) tracking interval can be set independently from online tracking interval, so you could for example send infrequent online updates, while still keeping a detailed local log.
  • OsmAnd also supports OsMo, a Russian service linked to the open source OsMoDroid GPS tracker for Android (Github), but I did not test that.

Home page: http://osmand.net/

Play store: https://play.google.com/store/apps/details?id=net.osmand

 

 

Other solutions

Even though there are plenty of GPS trackers on the market these days, many of which support some kind of export to online services, or even live updates, only very few have the option of a self-hosted server. I guess one of the reasons is, that most apps that support live updates are designed for a specific service (examples: Glympse, InstaMapper, LocaToWeb, guibber), where monetizing the users is the main goal. Having an open API or supporting self-hosted solutions is not in their interest. As a matter of fact, I haven’t been able to find any other solution in addition to the ones discussed above.

UPDATE 10 february 2015:

The other day, I found Gps Tracker for WordPress, which combines a WordPress plugin and an Android app to do more or less what I am trying to achieve here, except it focusses on tracking of multiple people/devices in a single map view. The Android app seems to send at most one location update per minute, and as such it’s not very suitable for sports and activity tracking. The map view in the WordPress plugin also doesn’t really draw a track (line), but merely a collection of waypoints that form the location history of the person/device you’re tracking. This app is best compared to Owntracks (reviewed above), because its goals seem similar.

(end update)

There is just one more possibility I can think of, but it’s not very practical and it requires a rooted phone. Just follow this five-step program:

  1. take one of the apps that uses a hosted service for live tracking
  2. find out the hostname / URL to which live updates are posted
  3. divert the hostname of the service to your own server via /etc/hosts (requires root on Android)
  4. reverse-engineer the protocol
  5. enjoy!

If you have other ideas, or apps to add to this page, please please please let me know!

Tags: android, GPS, maps, oruxmaps, osmand, owntracks, software, tracking, trackme
« 1 … 3 4 5 6 7 … 27 »

Today's motto

Do it, and call me when it’s done!

Categories

Recent Posts

  • Web maps in 2025
  • Adele in Munich
  • Trackserver v5.0 released
  • Trackserver v4.0 released
  • Panoramas Chamrousse

Meta

  • Log in
  • Entries feed
  • Comments feed
  • WordPress.org

Tags

aeroquad android arduino backports camera cat debian Digikam docker garmin Google GPS gpsbabel KDE kernel launchpad linux lowlands maps multicopter Natty oracle oruxmaps osmand owntracks pannellum phishing photography photo sphere pi plugin policykit quadcopter raspberry raspberry pi shield software spam tracking trackme Ubuntu viking wheezy wordpress Zümo

Copyright GRENDELMAN.NET 2025 | Theme by Theme in Progress | Proudly powered by WordPress