Showing posts with label NTP. Show all posts
Showing posts with label NTP. Show all posts

Saturday, January 05, 2013

Parsing NMEA sentences from GPS with Python + PySerial

I've had a need to parse some NMEA output on my Raspberry Pi for a project I'm working on. In essence, it is pretty trivial to read from a serial port and parse ASCII data in any programming language, but to build some resiliency and efficiency in need to be handled with some care.

I happen to interfacing with an EM-408 GPS module with my Raspberry Pi off the GPIO Rx/Tx USART GPIO pins.

If you need a quick reference for NMEA sentence standard, go here.

Working with PySerial


Below is a quick and dirty code sample to interface with a USART/serial interface. The biggest thing to take into consideration is the 'timeout' option when creating your serial.Serial() object.

From my trial and error process, specifying timeout=0 (e.g. no blocking at all), while makes some sense in a GPS NMEA sentence polling application to return immediately and keep reading output, it causes serious amounts of CPU overhead (almost 100% utilization).

Eliminating the timeout altogether (wait forever) isn't a great idea either because your code will endlessly block/wait for output from the GPS module; not good if the module ever dies/power loss/etc.

Setting a gracious timeout of 5-10 seconds (e.g. timeout=5 or timeout=10) seems to help out as well and end up being the best of both worlds.

Here's a snipit of my class for the EM-408:

import serial 

class EM408GPS:
    
    def __init__(self, serialport, baudratespeed):

        self.gpsdevice = serial.Serial(port=serialport, baudrate=baudratespeed, timeout=5)
        
        self.init()

    def init(self):
        
        if self.isOpen():
            return True
        
        return False

    def open(self):
        self.gpsdevice.open()
        
    def isOpen(self):
        return self.gpsdevice.isOpen()


That rough class sketch should be a perfect class wrapper to get you going with interfacing with a GPS via serial port or USART pins on the Pi.

Reading data with PySerial: Buffer or Newline?


This was the most interesting piece so far with. PySerial has a handful of methods for reading data that I tested with:

  • read(): This method reads the size of bytes from serial port input buffer.
  • readline(): This method reads serial port data down until a "\n" (newline) character is observed, then returns back a string.

    To be clever and witty, you'd generally want to use something like readline() since each NMEA sentence that it output to the serial port is terminated with a CRLF, right? I mean, why the hell wouldn't you? The answer is wrong the second you notice the very high CPU utilization happening when reading data.

    The good thing is this isn't a new problem, as it's a documented quite extensively on stack overflow amongst other places.

    The better way I found to attack this CPU utilization problem, is to take advantage of another method that PySerial offers:

  • inWaiting(): Return the number of bytes currently in the input buffer.

    ...and used this in combination with reading just '1' byte with read() then read whatever is left in PySerial's input buffer, then return for me to parse.

    Here's my class method called 'readBuffer()' partly solves this issue:

    def readBuffer(self):
    
            try:
                data = self.gpsdevice.read(1)
                
                n = self.gpsdevice.inWaiting()
                
                if n:
                    data = data + self.gpsdevice.read(n)
            
                return data
    
            except Exception, e:
                print "Big time read error, what happened: ", e
                sys.exit(1)
    

    The next part to deal with is now that we are reading everything out of the input buffer, our NMEA sentences aren't exactly in sentence order anymore.

    Now we have to leverage a bit of coding to properly find the start and end of a NMEA sentence. It's not too bad of an effort since we know a NMEA sentence starts with a '$' and ends with 'CRLF'. The key point is to find the CRLF in your read data buffer, then ensure to use the right end of that CRLF split (which is the start and some data of your other NMEA sentence) as the new start of the data buffer to construct the next line until you find the next CRLF, and so on...

    Here's the code snipit from my main() area that shows the initialization of the GPS and the read out of the NMEA sentences from my readBuffer() method:

    import re
    
    ...
    
    def main():
    
        device = EM408GPS("/dev/ttyAMA0", 4800)
    
        newdata = ""
        line = ""
    
        while device.isOpen():
             # If we have new data from the data CRLF split, then 
             # it's the start + data of our next NMEA sentence.  
             # Have it be the start of the new line
             if newdata: 
                 line = newdata
                 newdata = ""
             
             # Read from the input buffer and append it to our line 
             # being constructed
             line = line + device.readBuffer()
                
             # Look for  \x0d\x0a or \r\n at the end of the line (CRLF) 
             # after each input buffer read so we can find the end of our 
             # line being constructed
             if re.search("\r\n", line):
                 
                 # Since we found a CRLF, split it out
                 data, newdata = line.split("\r\n")
    
                 print "----" + str(datetime.datetime.now()) + "----"
                 print data
                 print
                        
                 # Reset our line constructer variable
                 line = ""
    


    Below is graphed output from 'vmstat' on the Raspberry Pi (in 2 second intervals) showing the performance benefit from using readBuffer() approach with read() + inWaiting() vs. using PySerial's readline():




  • Sunday, November 25, 2012

    Motorola Oncore UT+ Interface and PPS Breakout Board + Schematics

    I've been a long dabbler of all things 'network time' related for 5+ years now and do maintain a fair number of stratum-1 time sources for myself (two Motorola Oncore UT+, one Trimble Thunderbolt and soon-to-be WWVB). I shouldn't be that selfish and really make them a part of the NTP public pool; we'll see.
    Out of the several hobbyist-affordable time sources to obtain (Garmins or any NMEA-output), I do think the Motorola Oncore UT modules are pretty top-notch in terms of reliability, stability and performance. I think it provides just the right amount of challenge and barrier-to-entry for anyone who wants to dabble in EE and do some TTL-to-RS232 logic shifting to get one of these to work.

    Reasoning Behind This

    1) Arrangement and un-honkify my setup: To nicely arrange the Motorola Oncore module with a MAX232 and a battery for backup onto some sort of prototype perf board. For the LONGEST time (3+ years), I had an individual TTL-to-RS232 DB9 breakout screwed to some wood, and used a DB9-serial ribbon cable (since the Motorola Oncoreh has a 10-pin male header and the DB9-serial conveniently has a 10-pin female socket + ribbon). I'm no engineer, but I like to pretend to be one on my spare time (and at work when I'm allowed to) but I thought it was about time to make my stuff look a little bit more professional and less honky.
    2) No onboard battery for almanac storing with power-loss: My UT+ modules do NOT come with an onboard battery, thus every time I'd lose power, the cold-start time on the GPS was enough to piss me off from time to time. I used to have a UPS but kW usage over time really added unnecessary 'cost' to my electric bill to the point where I could really live without forking over several hundreds of dollars to 'the man' a year for the few times I lose power intermittently.
    3) Constant falseticker issues with current hardware setup and PPS from Motorola: For whatever reason (and I think partly due to newer ntpd versions, see this post) ntpd would label my PPS timesource as a falseticker all the time and it was frustrating. I didn't want to do some configuration hack and force truechimer status with 'true' in the ntp.conf config. What I was currently doing is sending the 1PPS TTL signal straight to DB-9 Pin 1 (carrier Detect) on the assumption that RS-232 logic is 3-24v, so clearly +5v TTL would 'barely' fall into it that logic path, but would. After some forum posting, I decided to move the 1PPS signal through the MAX232 TTL-to-RS232 shifter as well and my current circuit only took two inputs; one for Rx and Tx. So a redesign was needed.
    4) Get the RS232 DB-9 serial output and antenna to be in the same location: Right now I plug my DB-9 serial cable in on the right side then the antenna is on the opposite side so it makes positioning this thing in my closet hard with my crappy setup.

    The 'Solution' (so far)


    This certainly isn't complicated at all. Biggest time waster on this is:
    1) Soldering time --- take a while to solder all that together when it's thru-hole.
    2) DESIGN!!!: Because I wanted this to 'snap' together and sit on the underside of the UT module, I had to really think about this. I certainly didn't plan it out as nice as I turned out (I think). Here's another few pictures of me putting it together, along with the final product, and how it fits together underneath the Motorola Oncore module:

    Parts List

    Since I did my schematic with Digikey's scheme-it, I was able to make a nice bill of materials in case someone else wants to dupe this. I'm not endorsing Digikey, but you'll be able to see what I used; I had all these parts on hand from years of accumulation.

    Object  Name            Value           Digikey Part #
    R1      RESISTOR        1k8              CF14JT1K80CT-ND
    R3      RESISTOR        470k             CF12JT470KCT-ND
    R4      RESISTOR        330k             CF18JT330KCT-ND
    J2      2X5                              952-1924-ND
    J3      9-PIN,          FEMALE           A32510-ND
    IC1     IC              MAX232CPE        MAX232CPE-ND
    C1      NON             1uf              P4537TB-ND
    C2      NON             1uf              P4537TB-ND
    BT1     BATTERY         HOLDER CR2032    BU2032SM-HD-GCT-ND
    J4      1X2                              TMS-104-02-S-S-RA-ND
    C3      NON             1uf              P4537TB-ND
    C4      NON             1uf              P4537TB-ND
    R2      RESISTOR        1k8              CF14JT1K80CT-ND
    L1      LED             Red              751-1089-ND
    L2      LED             Green            51-1105-ND
    
    Get whatever type of thru-hole prototype perf board you want; I had some leftover stock I picked up from Radioshack a long time ago.

    Improvements To Add

    1) Put some inline diodes for some circuit protection along with a 5v voltage regulator --- I'm under some wild assumptions that I'm the only one who will ever use this and always apply 5v DC supply to it.
    2) Still work on that battery indicator just because after I stuff this up in my closet, it gets easily forgotten. I stumbled across this for a 3v low battery flasher using a Panasonic MN13811-G, but that MN13811 was pretty hard to find anywhere (digikey, mouser, etc.)
    3) Get it off the wood 1x4 mount and get a proper project box to enclose this.

    NTPd info

    Here's a snapshot of my tuned ntp.conf for the Motorla Oncore UT's (in case you need a jumpstart on it):

    enable pps 
    pps /dev/oncore.pps.0 hardpps
    
    # Oncore Motorola UT+
    server 127.127.30.0 minpoll 6 prefer
    fudge 127.127.30.0 time1 0.1988
    
    # Oncore Motorola UT+ 1PPS on DCD
    server 127.127.22.0 minpoll 6 prefer
    fudge 127.127.22.0 time1 0.39726
    
    tos mindist 0.010
    
    Here's a look at my peers listing:

    $ ntpq -p
    remote           refid   st t when poll reach   delay   offset  jitter
    ======================================================================
    LOCAL(0)        .LOCL.   8 l  486   64  200    0.000    0.000   0.000
    *GPS_ONCORE(0)   .GPS.   0 l    3   16  377    0.000    0.010   0.003
    oPPS(0)          .PPS.   0 l   34   64  377    0.000    0.099   0.133
    $
    

    Resources

    A few sites of interest I referenced and used for the idea:
    1) TAPR Motorola Oncore Interface SemiKit (http://www.tapr.org/gps_vpib.html): I'm sure I've looked at this while re-referencing the Motorola Oncore manuals from here, but this is EXACTLY what I wanted to do. However, I'm not going to pay to get boards made, nor do I want to have an all-fitting Motorola Oncore interface board; I just have UT's.
    2) Simple low battery indicators (http://kyorune.com/modding/article.php?id=55): This is very, very barebones. However, it gave me a good opportunity to do some testing, even though I kind of knew a head of time this was going to be too expensive milli-amp wise on the battery to even make it useful. The Motorola draws in micro-amps, so the the added value of an LED drawing at milli-amps just makes no sense.