Damir Systems Inc.

Google's gtalk is available inside a browser (chat only) or as a separate desktop application (chat + voice). If you have a gmail or Google Apps account you are good to go.

For the example I needed two accounts — was not able to send messages from "me" to "me".  The code does not check for availability of the recipient, so make sure that you are logged to the gmail or have the gtalk running.

The three basic modules:

• template,
• authorization,
• service;

To select the service in the main gps_event.py module:

Finally, the result: 

Another notification service, the example actually uses Google's Gmail. There is no need for an extra email account, because you can send email from "you" to "you" — as in the example. Similar to the previous post, we need three new modules:

• template,
• authorization,
• service;

and to select the service in the main gps_event.py module.

The sample delay is set to three minutes, not to overload  the inbox.

To be continued.

"When this happens, notify me." The basic expectation one could have from an EP application. Once upon a time, automated pager notification systems would cost a fortune, but times have changed for the better. Here is what it takes for our event (see pervious posts) to show up on twitter.

First we need a twitter account and a desktop client. It takes less than five minutes to get both of these, so just do it.

For this example I needed two accounts, one for me and the other one for the event generator (GPS). Instead of creating a new account for the GPS, I have simply borrowed my cat's account — just for the experiment.

Time for all that refactoring from the previous post to pay off; only four steps in adding a service now:

1. Add template module;
2. add authorization module;
3. add service module;
4. select the service in the gps_event.py.

Twitter is a short message service, so I have selected only: entity, time_stamp, latitude and longitude for this template.

Authorization data.

Service module sends messages.

Select 'tweet' as a service and 180 seconds for the delay in the gps_event.py. Twitter currently limits hourly number of request to about 40, so we'll be at half of that.

Those following phoebe_cat on twitter may have seen tweets like this one lately.

To be continued.

Time to clean up a bit. As new services are added this code is becoming bulky. I was writing it bottom-up, code as you go — no wonder. Time to stop coding and reflect on it, before it bloats out of control.

The first task is to identify code that changes with the addition of a service.

It should be noted that — at present — multiple templates are loaded in memory, but only one is needed.  This will have to be reworked to load code and a template for only one service at a time.

So here is the task list:

• Extract GPS data collection and parsing into a separate module or class.
• Each template gets its own module with generic template() function to return the string.
• Put each service in its own module with generic send()  function.
• Load required template and service modules dynamically into the main module.
• Try to follow Python naming conventions for function names, classes etc.
• Use consistent naming convention for module files.
• And first, before anything else, write a test set (unit test) to deal with bugs introduced while refactoring.

Time goes by.

Ok, done; these are new (modified) modules, after refactoring.

Services:

• gps_service_none.py — no service, print to screen only;
• gps_service_aws.py — Amazon SQS bus;
• gps_service_rc.py — ruleCore Geofence CEP;
• gps_service_tweet.py — Tweeter messages;
• gps_service_coral8.py — Coral8 ESP/CEP;
• gps_service_email.py — plain email.

Templates:

• gps_template_xml.py — generic XML;
• gps_template_json.py — generic JSON;
• gps_template_rcxml.py — ruleCore XML;
• gps_template_tweet.py — short message for Twitter;
• gps_template_email.py — format for email.

Authorization data:

• gps_auth_aws.py — Amazon SQS;
• gps_auth_rc.py — ruleCore;
• gps_auth_tweet.py — Tweeter;
• gps_auth_coral8.py — Coral8;
• gps_auth_email.py — plain email.

Other:

• gps_event.py — main module to run;
• gps_devices.py — collecting and parsing GPS data;

Lots of files, but keep in mind that only one set (service  + template + authorization) is loaded at a time. All files reside in the same directory. I do not list test units here, essentially each code module has a matching test_.py. You may notice that some services are not implemented yet, the modules are simply place-holders. This is one of the benefits of refactoring, it does not just clean up the code — it includes redesign to allow for seamless future changes.

With this structure in place I should be ready for the next fun episode — twitter the GPS.

To be continued.

Marco graciously offered me a temporary tinker-a-bit account on the ruleCore Geofence service, so I will try to move my event objects (see previous posts) to their server and see how it goes. Again, this is a blog-as-you-go series and at this time I have only a fuzzy idea of what lies ahead.

The ruleCore API seems straightforward, I like that.

Each event has its own predefined XML structure; at present I am interested only in the location event. Here is an example from the API:

It obviously looks different from my event-object format:

Time to summarize the difference in data sets.

		Comment
id	eventId	Ok, both are UID strings.
type	--	GPS is default for the service.
time_stamp	eventTimestamp	Ok, same format.
entity	EntityId	Ok, both are strings.
latitude	--	Combined as decimal format in coordinates (mine is deg min).
longitude	--	Combined as decimal format in coordinates.
speed	n/a	Not in ruleCore format.
course	n/a	Not in ruleCore format.
valid	n/a	Not in ruleCore format, have to make sure not to transmitt invalid records.
--	Coordinates	Combines latitude, longitude and altitude in decimal formats. I do not have altitude; mine is marine format, always zero.
n/a	Credentials	I do not have this; will be part of the template.

Not bad, almost everything is here — it just needs a new template and search/replace will do the rest.

The ruleCore service requires ordered events. The Amazon SQS bus (see the previous post) does not guarantee ordered arrival of messages, hence I will have to transfer data directly to ruleCore.

Once I get events over to the server I will think about setting up zone rules and output events.

To be continued.

Disclosure:

Everything I write about ruleCore is my own opinion. There is no contract nor any kind of active business relationship between ruleCore and me (Damir Systems Inc). The following is from Marco's email:

... there's no need for an NDA, you can write whatever you like about ruleCore. ...

With the data bus in place, it is time to start moving event objects (see previous posts).

The first module — qps_aws_auth.py — contains only authorization data for the Amazon SQS service, which brings us to the first problem. One can have many queues registered, however there is only one user-password pair for all the queues and for all operations on these queues. This is an equivalent of having a database allowing for only one user with admin privileges — not quite designed for sharing.

Note:

A recent entry in the SQS forum indicates that Amazon is working on queue sharing between different AWS accounts, which would remedy the problem.

Boto library handles all ASW communications, so the code is short; name the module gps_aws.py.

Only sendToCep(xst) function in the gps_event.py module (see previous post) has to be modified.

This is all it takes to move our GPS events to a globally available message bus. Having a sensor-only device, we do not need to read from the bus in this application, but here is the read code for debug purposes. Note that Boto uses base64 encoding, so in order to exchange data between SQS Scratchpad and Boto a base64 encoder/decoder is needed, like this one.

Add the following code to the end of the qps_aws.py module.

Once in the bus, messages are available globally to any application which can communicate to Amazon SQS; libraries are available for almost all software platforms. Messages expire after four days.

The code should run unchanged on Windows, Linux and Mac — as long as your GPS transmits the $GPMRC NMEA message. Some experimentation may be needed with serial to USB converters for laptops without a serial port.

Coming up next: transfer to one of EP/CEP servers.

Now that I have some events flowing — see the previous post — it is time to route them somewhere. I was going to start working on Coral8, but this morning I saw this article on Amazon Simple Queue Service (SQS); so I thought, why not? I always wanted a message bus — hey, everyone should have one; holiday season and all.

Here is what it takes to get a message queue these days:

Step 1 - administer:

• Sign up for an Amazon AWS account.
• Sign up for the SQS service — this is where you need your Visa card.
• Obtain AWS access keys from 'Your Account' menu on the AWS services page.
• Download the Amazon SQS Scratchpad so you can start tinkering immediately.

Step 2 - set it up:

• Start the SQS Scratchpad.
• Enter access keys.
• Select Create Queue from the Explore API list.
• Fill in the Queue Name and leave the Visibility Timeout blank — default is 30seconds.
• Click Invoke Request, the service returns the queue URL — jot this down.
• It takes some time for this request to propagate through AWS, so have a coffee — tea will do too.

Step 3 - tinker:

• Select List Queues from the Explore API list;
• click Invoke Request;
• the service responds with something like this:

And that's it, wasn't bad at all. There is also a nice manual to guide you along the way.

Now you can try writing to and reading from the queue — use the SQS Scratchpad. The Scratchpad exposes the SQS API, so it is possible to manually explore the API commands before one starts programming. Libraries are available for Java, PHP, VB.NET, C#, Perl, Python.

Here are a few things one learns very soon:

• Messages are stored redundantly across several servers — that's nice.
• Maximum number of messages that can be returned in one read is 10.
• If the queue has less than 1000 messages, the read service may actually return less than 10.
• If the queue has only few messages, the read service may return none of them — so the read has to be repeated for service to scan different servers.
• The order of messages is not preserved.
• According to the manual, it is possible to receive a message more than once if all the servers containing the message are not communicating at the time of delete-message operation.

For example — regarding the order of messages — I wrote 10 messages to the queue [Msg 0, Msg 1,..., Msg 9] and read them back.

In each read attempt I was asking for 10 messages, the maximum number. Is this good or bad? Depends on the application, I guess.

One more detail, when exploring the SQS Scratchpad use Firefox or IE, Google Chrome does not seem to render XML trees nicely.

Next step is to use Python library and start transferring GPS data to the SQS queue.

To be continued.

After some tinkering I was able to cook up some Python code to get the GPS data. So, a Magellan Meridian Gold GPS is connected over the serial port to a PC running Windows XP and Python. The Python version I am using is 2.52 and if you want to play with the code you may want to install the pySerial library before starting.

First, here is the result—cut & paste—from Python Shell.

The program reads the serial port every 10-12 seconds, extracts the position and speed data from a $GPRMC message and packages the data into XML form of an event object.

The Python code has two modules (files), the first one— gps_event_template.py —contains only a template for XML and JSON strings; the main module— gps_event.py —has all the working code.

The template has placeholders—like $time_stamp_here$ —which are simply later replaced by GPS data.

Here is the main module; if running from Python Shell, start this one. Python is an easy read, so you should not have trouble following. The sendToCep(xst) function simply prints the XML to the Shell; I need to tinker with the CEP server before implementing this one. Keep in mind that this is a blog-as-you-go project.

So the next step is a CEP server—install, configure, communicate, do something with.

To be continued.