Adding 'Hack' monospace font functionality into Spyder IDE in Fedora Core 21

As of late, a lot of the code I have been hacking out for just about anything has been straight, 100% Python.  With that, I've also ditched 'vi' + plugins and have grown extremely comfortable (and probably complacent?) using Spyder IDE GUI.  I never wanted to be GUI dependent (again) but I am, so whatever.  I deal with it.

But, recently, I saw a post on arstechnica.com touting the coolness of Hack, an open-source monospace font marketed towards the code slingers of the world.

I undoubtedly get tired of using of the defacto DejaVu and Liberation monospace fonts and decided to give this a shot.  And quite honestly, even though it's subtle, I really like it.

Characters of note:  I don't notice the lowercase-L vs. the numeral 1 as much as I do the zero vs. uppercase-O and lowercase-I curvature.

Font sizes:  I think my favorite font size is 9px or 11px (11px especially if I've been staring at my monitor for more than 4 hour straight with contacts in to boot!)

The overall font character scheme also seems a bit more compressed as well, which is nice so I 'feel' like I'm getting more width real-estate.  Case in point:  take it for what it's worth.

Even though this is a less traditional and definitely 'non-distro' way to add an overall font to your OS, I think it gives most people the flexibility enough to give it a shot themselves with as minimal overhead as possible, especially if you want it 'long-term' and want to put it in the same place as your other distro fonts via package management without creating a distro package for it.

Here's how to make the 'Hack' monospace font accessible in Spyder IDE under Fedora (or any other RHEL based OS for that matter).

If you really want to get technical and overview-ish, I guess this would be a good way to hack-and-slash 'any' font you want to Spyder, monospace or not.

Download and Unzip 'Hack' Font from SourceFoundary

Go to Hack's SourceFoundary site and download the True-Type font .zip archive.

Once you have downloaded the 'Hack' fonts, simply unzip them wherever you want:

$ cd /path/to/hack/fonts/
$ unzip Hack-v2_010-ttf.zip
unzip Hack-v2_010-ttf.zip 
Archive:  Hack-v2_010-ttf.zip
  inflating: Hack-Bold.ttf           
  inflating: Hack-BoldOblique.ttf    
  inflating: Hack-Regular.ttf        
  inflating: Hack-RegularOblique.ttf
$

Add 'Hack' font into to your OS

Adding the Hack monospace font (this way) is pretty trivial.  The key point here is to know the top-level path where your Linux distro stashes it's fonts.  For a font like this that I really want to use long-term, I think this is a better solution than putting it in ~/.fonts/  or equivalent. I'm also going to assume you have proper privileges to do all these command(s) or know how to obtain it (e.g. NOT prefacing the sudo command to obvious root-level stuff)

• Create a new directory for the 'Hack' font in your Linux distro's top-level system font path

$ mkdir /usr/share/fonts/hack

• Copy fonts over

$ cp -a /path/to/hack/fonts/Hack-*.ttf /usr/share/fonts/hack/

• Make sure permissions + ownership are sane on your font directory and fonts

$ chown -R root:root /usr/share/fonts/hack && chmod 755 /usr/share/fonts/hack

Changes to Spyder IDE

I'm using Spyder IDE v2.2.4, so YMMV if you're using another version.  I didn't go scout all this out, but it's pretty self-explanatory.

Spyder IDE has a default, hard-coded list of approve and available monospace fonts that are defacto standards.  All you have to do is alter the list data structure defining the list of monospace fonts in spyderlib.

To do this, you'll want to look for the following line of code in /usr/lib/python2.7/site-packages/spyderlib/config.py module:

MONOSPACE = ['Monospace', 'DejaVu Sans Mono', 'Consolas', 'Monaco',
                'Bitstream Vera Sans Mono', 'Andale Mono', 'Liberation Mono',
                'Courier New', 'Courier', 'monospace', 'Fixed', 'Terminal']

...and add in the 'Hack' fonts into the MONOSPACE list structure so it looks something like this:

MONOSPACE = ['Monospace', 'DejaVu Sans Mono', 'Consolas', 'Monaco',
               'Bitstream Vera Sans Mono', 'Andale Mono', 'Liberation Mono',
               'Courier New', 'Courier', 'monospace', 'Fixed', 'Terminal',
               'Hack-Regular', 'Hack-RegularOblique.ttf', 'Hack-Bold.ttf',
               'Hack-BoldOblique.ttf']

NOTE:  There are several 'config.py' modules in Spyder IDE, so if you're really confused at this point, just do a case-insensitive recursive grep (e.g.  grep -i -R "MONOSPACE" /path/to/spyderide/site-package/install) and find the right one to edit.

Restart Spyder IDE and Choose 'Hack' as your Font

If you still have Spyder IDE running, close it and re-open it.

Then on Sypder's top menu bar, you'll want to navigate to 'Tools > Preferences', then select the 'Editor' option on the left-hand side.  Now, if you select the 'font' drop-down menu, you should see 'Hack' available for your choosing:

You can now click 'Apply' and watch all those wonderful tabs of 'code' change over to using 'Hack'.

If you also want your Python console in Spyder to use this font as well, just select 'Console' and change it there, too.

Before-and-After of Hack vs. DejaVu Sans

Here's a couple of quick screenshots put in extra-large to show the differences of Hack vs. DejaVu Sans out-of-the-box Spyder IDE font:

DejaVu Sans - Normal - 9px

Hack - Normal - 9px

Now as much as I'd want to package this into an RPM for my own sake in case some update comes about, that would just be more work than it's worth.

Enjoy!

1PPS Support with GPIO For Raspberry Pi in Fedora 17 Remix

One of my projects as of the last 5 months (although slow going) is making a full-out stratum-1 time server appliance out of the Raspberry Pi.  I'm going to be using an EM408 GPS module + generating my own 1PPS signal with an AVR attiny84 microcontroller inputting into GPIO on the Pi, and if i have enough GPIO left, will be driving a 16x4 LCD parallel output-only display for fun as well.

When it comes to just using the Raspberry Pi as a NTP stratum-1 server, there's quite a few documented artifacts out there on the internets already being done, most notably from David Taylor's excellent documentation on his website.  

It's worthy to note that most, if not all, Raspberry Pi Linux distribution spins DO NOT come with CONFIG_PPS or CONFIG_PPS_CLIENT_GPIO compiled into the kernel or as modules.  

I'm not totally down with the amount of hackery floating around to insert dirty modules into the kernel.  Ya, it works.  Yes, if you have  CONFIG_MODVERSIONS set, then you should be able to install modules not compiled with your kernel into it.  However, I'm rolling with Fedora 17 remix and option isn't compiled in with the stock kernel.  So either way, I had to result to building my own RPMs for this.  

If you're interested in getting 1PPS supprt on the Raspberry Pi via GPIO (like I and many other time enthusiasts are), then here's some steps to get you in the right direction.  Below are some options you have:

Download 1PPS Packaged RPM for Fedora 17 Remix

Here's a link to my binary RPM package for 1PPS, which includes support for:

• PPS Support (CONFIG_PPS=m)
• PPS Source using GPIO pin (CONFIG_PPS_CLIENT_GPIO=m)

Download from: http://www.wisehippy.com/files/

NOTE:  Yes, I am using the FC 18 source rpm, but they will install on FC 17.  I have tested.

Build your own Kernel + RPMs

There's quite a few step(s) I'll omit since it's clearly beyond the scope of this blog post, but here's a quick and dirty run-down on how to do it on an RPM-based Linux distribution:

1)  Build yourself a soft-float ARM cross compiler (at least for Fedora 17 remix) using crosstool-ng

2)  Set up your RPM build environment using `rpmdev-setuptree`

3)  Download the raspberry-pi-kernel source RPM from here

4)  Patch the kernel config ( in SOURCES folder) and .spec file (in SPECS folder) using `patch` with my patches that are posted on the Fedora Remix as an 'enhancement' option:  https://fedorahosted.org/arm/ticket/64

5)  Rebuild kernel package + sub-packages with:  

# rpmbuild -bb --clean --target=armv5tel raspberrypi-kernel.spec

6)  Copy 'at least' the kernel-pps RPM package to you RPi and install with:

# rpm -ivh raspberrypi-kernel-pps-3.2.27-1.20120926git9245b4c.rpfr18.armv5tel.rpm

7)  On the RPi, run:

# modprobe pps-gpio

     ...then verify the module loaded correctly with:

# dmesg

     ...and look for some output like:

[ 1404.141098] pps_core: LinuxPPS API ver. 1 registered
[ 1404.141123] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti 

Do note that applying my patches in step #4 will also apply some of my other .spec file changes that change the way the kernel image is built and installed on the FC remix distro.  Do take a look at the changes and don't apply blindly if you can help it.  This only matters if you plan to install my kernel RPM.

Enjoy!

Calculating NMEA sentence checksums with Python

Another feature I wanted to add to my NTP/GPS project I'm doing with my Raspberry Pi is to validate the NMEA sentences I'm reading in by checksum'ing the NMEA sentence and comparing it with the XOR checksum provided from the GPS receiver.

This is especially important if you plan on writing an application that is reading $GPGGA, $GPGLL or $GPRMC sentences for timekeeping purposes and want to validate the sentence you read for time is legit or not.

Background on NMEA Checksums

After reading some documentation on NMEA sentence types and, more importantly, about checksums, the process is actually quite easy and can be coded up fairly easily.

Here's the main blurb from the NMEA documentation about the checksum and how to calculate it:

Programs that read the data should only use the commas to determine the field boundaries and not depend on column positions. There is a provision for a checksum at the end of each sentence which may or may not be checked by the unit that reads the data. The checksum field consists of a '*' and two hex digits representing an 8 bit exclusive OR of all characters between, but not including, the '$' and '*'. A checksum is required on some sentences.

Not so hard, right? We need to exclusive OR (XOR) all of the characters (INCLUDING the commas) between the '$' and the '*'. 

Let's do it!

NMEA Sentence Breakdown

Here's a NMEA sentence example that I will use to show the XOR process on:

$GPGGA,174134.000,4345.9112,N,09643.8029,W,1,05,2.7,452.6,M,-27.1,M,,0000*60

The color breakdown is:

            Black: Two positional characters in the NMEA sentence we need to read our characters 
                      from between
            
            Green:  Characters we need to XOR
            Blue:  The calculated checksum we need to compare our calculated checksum against

Bitwise XOR on NMEA Sentences

As the documentation states, we need to exclusive OR (XOR) all of the characters (INCLUDING the commas) between the '$' and the '*'.

The process is quite simple: We want to take each character and XOR it with the previous XOR'd output from the last character. The very last character that is XOR'd will be the final checksum value that you'd then compare with the checksum value.

If you're a bit fuzzy on on XOR, the bitwise operator in most programming languages is ' ^ ' and the rules are as follows:

• 1 ^ 1 = 0
• 1 ^ 0 = 1
• 0 ^ 1 = 1
• 0 ^ 0 = 0

One thing to note, when you start of XOR'ing, you'll want to compare your first character with zero (e.g. 0, 0xFF, 0b0, etc.) so on the next character iteration (which would be the 2nd character in the NMEA string) will XOR against the binary value of the first character.  Why is this so?  If you look at the XOR rules above, we only need one bit 'on' (or '1') on to bit-flip-it and keep it 'on' (or '1').  So if we XOR against '0', we get our original value back.

Let's use our example sentence above to go through a handful of binary XOR iterations of what we'll be accomplishing in code:

0b0000000     0
0b1000111     G
----------
0b1000111     XOR output
0b1010000     P
----------
0b0010111     XOR output
0b1000111     G
----------
0b1010000     XOR output
0b1000111     G
----------
0b0010111     XOR output
0b1000001     A
----------
0b1010110     XOR output
0b0101100     ,
----------
0b1111010     XOR output
0b0110001     1
----------
0b1001011     XOR output

...

0b1010000     XOR output
0b0110000     0
----------
0b1100000     XOR output
0b0110000     0
----------
0b1010000     XOR output
0b0110000     0
----------
0b1100000     Our checksum (96 decimal, 0x60 HEX)

As you can see, we ended up with '0x60' which is the same as our example sentence above of '*60'. So we were able to validate this NMEA sentence!

NMEA Checksum Python Code

Now that we got the explanation out of the way, let's look at the code.  It's really simple:

def chksum_nmea(sentence):
    
    # This is a string, will need to convert it to hex for 
    # proper comparsion below
    cksum = sentence[len(sentence) - 2:]
    
    # String slicing: Grabs all the characters 
    # between '$' and '*' and nukes any lingering
    # newline or CRLF
    chksumdata = re.sub("(\n|\r\n)","", sentence[sentence.find("$")+1:sentence.find("*")])
    
    # Initializing our first XOR value
    csum = 0 
    
    # For each char in chksumdata, XOR against the previous 
    # XOR'd char.  The final XOR of the last char will be our 
    # checksum to verify against the checksum we sliced off 
    # the NMEA sentence
    
    for c in chksumdata:
       # XOR'ing value of csum against the next char in line
       # and storing the new XOR value in csum
       csum ^= ord(c)
    
    # Do we have a validated sentence?
    if hex(csum) == hex(int(cksum, 16)):
       return True

    return False

There you have it!

Parsing NMEA sentences from GPS with Python + PySerial

I've had a need to parse some NMEA output on my Raspberry Pi for a project I'm working on. In essence, it is pretty trivial to read from a serial port and parse ASCII data in any programming language, but to build some resiliency and efficiency in need to be handled with some care.

I happen to interfacing with an EM-408 GPS module with my Raspberry Pi off the GPIO Rx/Tx USART GPIO pins.

If you need a quick reference for NMEA sentence standard, go here.

Working with PySerial

Below is a quick and dirty code sample to interface with a USART/serial interface. The biggest thing to take into consideration is the 'timeout' option when creating your serial.Serial() object.

From my trial and error process, specifying timeout=0 (e.g. no blocking at all), while makes some sense in a GPS NMEA sentence polling application to return immediately and keep reading output, it causes serious amounts of CPU overhead (almost 100% utilization).

Eliminating the timeout altogether (wait forever) isn't a great idea either because your code will endlessly block/wait for output from the GPS module; not good if the module ever dies/power loss/etc.

Setting a gracious timeout of 5-10 seconds (e.g. timeout=5 or timeout=10) seems to help out as well and end up being the best of both worlds.

Here's a snipit of my class for the EM-408:

import serial 

class EM408GPS:
    
    def __init__(self, serialport, baudratespeed):

        self.gpsdevice = serial.Serial(port=serialport, baudrate=baudratespeed, timeout=5)
        
        self.init()

    def init(self):
        
        if self.isOpen():
            return True
        
        return False

    def open(self):
        self.gpsdevice.open()
        
    def isOpen(self):
        return self.gpsdevice.isOpen()

That rough class sketch should be a perfect class wrapper to get you going with interfacing with a GPS via serial port or USART pins on the Pi.

Reading data with PySerial: Buffer or Newline?

This was the most interesting piece so far with. PySerial has a handful of methods for reading data that I tested with:

read(): This method reads the size of bytes from serial port input buffer.

readline(): This method reads serial port data down until a "\n" (newline) character is observed, then returns back a string.

To be clever and witty, you'd generally want to use something like readline() since each NMEA sentence that it output to the serial port is terminated with a CRLF, right? I mean, why the hell wouldn't you? The answer is wrong the second you notice the very high CPU utilization happening when reading data.

The good thing is this isn't a new problem, as it's a documented quite extensively on stack overflow amongst other places.

The better way I found to attack this CPU utilization problem, is to take advantage of another method that PySerial offers:

inWaiting(): Return the number of bytes currently in the input buffer.

...and used this in combination with reading just '1' byte with read() then read whatever is left in PySerial's input buffer, then return for me to parse.

Here's my class method called 'readBuffer()' partly solves this issue:

def readBuffer(self):

        try:
            data = self.gpsdevice.read(1)
            
            n = self.gpsdevice.inWaiting()
            
            if n:
                data = data + self.gpsdevice.read(n)
        
            return data

        except Exception, e:
            print "Big time read error, what happened: ", e
            sys.exit(1)

The next part to deal with is now that we are reading everything out of the input buffer, our NMEA sentences aren't exactly in sentence order anymore.

Now we have to leverage a bit of coding to properly find the start and end of a NMEA sentence. It's not too bad of an effort since we know a NMEA sentence starts with a '$' and ends with 'CRLF'. The key point is to find the CRLF in your read data buffer, then ensure to use the right end of that CRLF split (which is the start and some data of your other NMEA sentence) as the new start of the data buffer to construct the next line until you find the next CRLF, and so on...

Here's the code snipit from my main() area that shows the initialization of the GPS and the read out of the NMEA sentences from my readBuffer() method:

import re

...

def main():

    device = EM408GPS("/dev/ttyAMA0", 4800)

    newdata = ""
    line = ""

    while device.isOpen():
         # If we have new data from the data CRLF split, then 
         # it's the start + data of our next NMEA sentence.  
         # Have it be the start of the new line
         if newdata: 
             line = newdata
             newdata = ""
         
         # Read from the input buffer and append it to our line 
         # being constructed
         line = line + device.readBuffer()
            
         # Look for  \x0d\x0a or \r\n at the end of the line (CRLF) 
         # after each input buffer read so we can find the end of our 
         # line being constructed
         if re.search("\r\n", line):
             
             # Since we found a CRLF, split it out
             data, newdata = line.split("\r\n")

             print "----" + str(datetime.datetime.now()) + "----"
             print data
             print
                    
             # Reset our line constructer variable
             line = ""

Below is graphed output from 'vmstat' on the Raspberry Pi (in 2 second intervals) showing the performance benefit from using readBuffer() approach with read() + inWaiting() vs. using PySerial's readline():

Motorola Oncore UT+ Interface and PPS Breakout Board + Schematics

I've been a long dabbler of all things 'network time' related for 5+ years now and do maintain a fair number of stratum-1 time sources for myself (two Motorola Oncore UT+, one Trimble Thunderbolt and soon-to-be WWVB). I shouldn't be that selfish and really make them a part of the NTP public pool; we'll see.
Out of the several hobbyist-affordable time sources to obtain (Garmins or any NMEA-output), I do think the Motorola Oncore UT modules are pretty top-notch in terms of reliability, stability and performance. I think it provides just the right amount of challenge and barrier-to-entry for anyone who wants to dabble in EE and do some TTL-to-RS232 logic shifting to get one of these to work.

Reasoning Behind This

1) Arrangement and un-honkify my setup: To nicely arrange the Motorola Oncore module with a MAX232 and a battery for backup onto some sort of prototype perf board. For the LONGEST time (3+ years), I had an individual TTL-to-RS232 DB9 breakout screwed to some wood, and used a DB9-serial ribbon cable (since the Motorola Oncoreh has a 10-pin male header and the DB9-serial conveniently has a 10-pin female socket + ribbon). I'm no engineer, but I like to pretend to be one on my spare time (and at work when I'm allowed to) but I thought it was about time to make my stuff look a little bit more professional and less honky.
2) No onboard battery for almanac storing with power-loss: My UT+ modules do NOT come with an onboard battery, thus every time I'd lose power, the cold-start time on the GPS was enough to piss me off from time to time. I used to have a UPS but kW usage over time really added unnecessary 'cost' to my electric bill to the point where I could really live without forking over several hundreds of dollars to 'the man' a year for the few times I lose power intermittently.
3) Constant falseticker issues with current hardware setup and PPS from Motorola: For whatever reason (and I think partly due to newer ntpd versions, see this post) ntpd would label my PPS timesource as a falseticker all the time and it was frustrating. I didn't want to do some configuration hack and force truechimer status with 'true' in the ntp.conf config. What I was currently doing is sending the 1PPS TTL signal straight to DB-9 Pin 1 (carrier Detect) on the assumption that RS-232 logic is 3-24v, so clearly +5v TTL would 'barely' fall into it that logic path, but would. After some forum posting, I decided to move the 1PPS signal through the MAX232 TTL-to-RS232 shifter as well and my current circuit only took two inputs; one for Rx and Tx. So a redesign was needed.
4) Get the RS232 DB-9 serial output and antenna to be in the same location: Right now I plug my DB-9 serial cable in on the right side then the antenna is on the opposite side so it makes positioning this thing in my closet hard with my crappy setup.

The 'Solution' (so far)

This certainly isn't complicated at all. Biggest time waster on this is:
1) Soldering time --- take a while to solder all that together when it's thru-hole.
2) DESIGN!!!: Because I wanted this to 'snap' together and sit on the underside of the UT module, I had to really think about this. I certainly didn't plan it out as nice as I turned out (I think). Here's another few pictures of me putting it together, along with the final product, and how it fits together underneath the Motorola Oncore module:

Parts List

Since I did my schematic with Digikey's scheme-it, I was able to make a nice bill of materials in case someone else wants to dupe this. I'm not endorsing Digikey, but you'll be able to see what I used; I had all these parts on hand from years of accumulation.

Object  Name            Value           Digikey Part #
R1      RESISTOR        1k8              CF14JT1K80CT-ND
R3      RESISTOR        470k             CF12JT470KCT-ND
R4      RESISTOR        330k             CF18JT330KCT-ND
J2      2X5                              952-1924-ND
J3      9-PIN,          FEMALE           A32510-ND
IC1     IC              MAX232CPE        MAX232CPE-ND
C1      NON             1uf              P4537TB-ND
C2      NON             1uf              P4537TB-ND
BT1     BATTERY         HOLDER CR2032    BU2032SM-HD-GCT-ND
J4      1X2                              TMS-104-02-S-S-RA-ND
C3      NON             1uf              P4537TB-ND
C4      NON             1uf              P4537TB-ND
R2      RESISTOR        1k8              CF14JT1K80CT-ND
L1      LED             Red              751-1089-ND
L2      LED             Green            51-1105-ND

Get whatever type of thru-hole prototype perf board you want; I had some leftover stock I picked up from Radioshack a long time ago.

Improvements To Add

1) Put some inline diodes for some circuit protection along with a 5v voltage regulator --- I'm under some wild assumptions that I'm the only one who will ever use this and always apply 5v DC supply to it.
2) Still work on that battery indicator just because after I stuff this up in my closet, it gets easily forgotten. I stumbled across this for a 3v low battery flasher using a Panasonic MN13811-G, but that MN13811 was pretty hard to find anywhere (digikey, mouser, etc.)
3) Get it off the wood 1x4 mount and get a proper project box to enclose this.

NTPd info

Here's a snapshot of my tuned ntp.conf for the Motorla Oncore UT's (in case you need a jumpstart on it):

enable pps 
pps /dev/oncore.pps.0 hardpps

# Oncore Motorola UT+
server 127.127.30.0 minpoll 6 prefer
fudge 127.127.30.0 time1 0.1988

# Oncore Motorola UT+ 1PPS on DCD
server 127.127.22.0 minpoll 6 prefer
fudge 127.127.22.0 time1 0.39726

tos mindist 0.010

Here's a look at my peers listing:

$ ntpq -p
remote           refid   st t when poll reach   delay   offset  jitter
======================================================================
LOCAL(0)        .LOCL.   8 l  486   64  200    0.000    0.000   0.000
*GPS_ONCORE(0)   .GPS.   0 l    3   16  377    0.000    0.010   0.003
oPPS(0)          .PPS.   0 l   34   64  377    0.000    0.099   0.133
$

Resources

A few sites of interest I referenced and used for the idea:
1) TAPR Motorola Oncore Interface SemiKit (http://www.tapr.org/gps_vpib.html): I'm sure I've looked at this while re-referencing the Motorola Oncore manuals from here, but this is EXACTLY what I wanted to do. However, I'm not going to pay to get boards made, nor do I want to have an all-fitting Motorola Oncore interface board; I just have UT's.
2) Simple low battery indicators (http://kyorune.com/modding/article.php?id=55): This is very, very barebones. However, it gave me a good opportunity to do some testing, even though I kind of knew a head of time this was going to be too expensive milli-amp wise on the battery to even make it useful. The Motorola draws in micro-amps, so the the added value of an LED drawing at milli-amps just makes no sense.

Brrrrrrrrrrrr! Are you Kidding me?!?!

The weather this year has been out-of-this-world COLD! Holy crap! I've been a resident of South Dakota all my life and it does get cold here, but down to an air temperature of -20 F?! I think I just said 'South Dakota' in that last sentence, NOT Siberia.


Above is a daily graph I make with `rrd-tool` using Dallas 1-wire DS18B20 sensors I have around my house and outside. I'm just absolutely blown away; I've seen combined temperatures and wind speeds (wind chills) reach -20 to -40 F before around here, but it's just hard re-call actual air temperature reaching that.

White-space removal goodness with Linux

I've been recently putting together a massive backup of all my digital wares on my network. Problem is, over the years, when I've converted any CD I had over to digital mp3's or made a copy of one of my DVDs, a lot of it was done without a lot of thought that I'd be backing it up to a UNIX box 9-10 years down the road; so a lot of the filenames by pure convenience to the application(s) I was using, uses white-spaces as delineations between fields on a filename.

Is that a real problem? Kinda because white-spaced named files are a hooker to work with under *NIX platforms. Case in point, it does cost me some sleep at night.

So where does Adam go when there's no where else to turn? the *NIX command-line.

sed example: sed -n "s/\s/_/g"

tr example: tr [:space:] _

...quick and dirty solution.

Fd Linux development starting again... finally

It's been a *really* long time (approximately 7 years) since I've worked on my OSS project, Fd Linux and I'm finally starting to get my lazy ass in check and getting a bit more free time on my week[days,nights,ends] to get my development box set up and work on a new version of Fd Linux.

It's kind been a quirky hiatus from this due to getting deployed to Iraq back in 2003 to 2005 and just getting back into the swing of things, along with getting a house, watching my two girls grow up, and been extremely busy with work. Life kind of gets in the way like that.

Obviously over the years, the PC market has moved away from floppy disks (I honestly think floppy disk drives are *extra* commodities to add to PC's instead of defacto standards) and with solid state media dirt cheap right now and well capable of holding more than 1.44MB's worth of Linux goodness, it's a no brainer to start moving towards USB flash drives.

Right now, I'm working on a side project that will help get the base started for Fd Linux development, which is a home-brew NTP server, running on a PCB-5825 (AMD Geode 300Mhz, 128mb PC-133), booting off of 256MB compact-flash and using a Motorola Oncore UT+ GPS module with PPS as it's time source. There's much more detail to that, but a lot of the rootfs generation groundwork from way back to Fd Linux v1.0.0 will be used to concoct a good setup for this project.

Any rate, stay tuned. I'm pretty excited to get back into my groove.

PAM max login limits in RedHat Enterprise 5

I found something surprisingly 'odd' with pam_limits.so library when setting up and base lining a few servers the other day at work.

For some reason you can not concurrently use both 'maxsyslogins' and 'maxlogins' at the same time in your /etc/security/limits.conf file to set max authenticated user/group logins per user or system. With my intentions of having a limits of anyone part of, lets say, the 'users' group, could not have more than 3 separate simultaneous logins into the box with a maximum hard limit of 15 total system logins just doesnt seem to work.

So if you're going to try something like this:

@users hard maxlogins 3
* hard maxsyslogins 15

...didn't work for me. I have noticed, also, if you have a particular user who is part of several groups (e.g. users and devel groups) and you have limits for both of those groups, once the first group limits is matched, it goes with that rule. But if you have any idea on how to get 'maxlogins' and 'maxsyslogins' to work together, I'd be happy to know.

Analyzing and cleaning out Linux/Unix filesystems that fill up

So I have this problem from time to time at work: bastid users or custom processes that either go rogue and start filling up filesystem space at an alarming rate, or crappy code that doesn't clean itself up.

Here's a couple of my favorite commands to help out with that:

`find` Command

If you aren't using `find`, you should. My favorite one-liners for stuff like this:

Find all files in a directory with a size between (for example) 1KB and 10MB that last changed 1 minute(s) ago:

[root@testbox]# find . -type f -size +1024k -and -size -10240k -cmin 1 -exec du -sb '{}' \;


Find all files in a directory with a size at least (for example) 75MB that last changed 1 minute(s) ago:

[root@testbox]# find . -type f -size +76800k -cmin 1 -exec du -sb '{}' \;


Find all files in a directory with a size between (for example) 1MB to 100MB, owned by 'frank' (UID: 501) that last changed 1 minute(s) ago:

[root@testbox]# find . -type f -size +1024k -and -size -102400k -uid 501 -cmin 1 -exec du -sb '{}' \;


...the execution of `du` in my commands will definitely add some overhead, so feel free to leave it off. It's a nice quick human readable presentation if you're in a hurry, but I wouldn't do it if you have a shitload of files that are going match your search.

`lsof` Command

Simply: list open files. This is your friend you can use in tandem with the `find` commands above to narrow down what process(es) are pillaging your filesystem.

What I do is take the output of a particular find command if I see growing files out there but cannot narrow down what process(es) it could be and just simply run:

[root@testbox]# lsof | grep file_name

...where obviously, file_name is some output you got from the `find` command(s).

Those are just some of many tricks to do to track down that type of ridiculousness that happens when things go bad late at night when you're on-call.

WEP pwn'age... what's next?

After all these years of known vulnerabilities with the technology, I'm still flippin' amazed the percentage of deployed, neighborhood 802.11b/g wireless access points that STILL have WEP encryption enabled. Not saying that WPA-TSIP-PSK is much better (well, if you've got a super, stellar dictionary/phrase list) but... what this oversight gives me is two things:

1) Entertainment for a very short period of time in 20 minute increments...
2) A very big 'network' to explore, play with and hide behind...

I can't give myself all the credit; `aircrack-ng` is a pretty slick tool.

However, just tonight, I've pwn'ed the last WEP-enabled access point on my block... well, all the ones I can reach without using my 20dbi gain omni antenna poking out my basement window ;-) ...now all that is left is burning my braincells watching television and reading slashdot.

Cisco VPN client and Fedora Core 8

I recently did a kernel upgrade on my laptop and noticed my Cisco VPN software wasn't working due to needing to recompile the Cisco `vpnclient` code to build the needed modules again.

...and I discovered two things: what I forgot and what I didnt forget.

1) Forgot: how fucking ridiculously broken kernel code is in Fedora's distribution stream. OMFG. Can a person compile anything outside hello_world.c against anything released? (harsh and far from true, but I'm pissed)

2) Didn't Forget: how painful it was to get `vpnclient` going again on my laptop the first time with all the patching and 32/64-bit specific OS crap to take into consideration.

Want the steps? Here you go.

Cisco VPN client packages and patches

I'm running Fedora 8 64-bit on my laptop, and this are the packages that worked for me. Get 'em:

• vpnclient-linux-x86_64-4.8.01.0640-k9.tar.gz
  
  
• vpnclient-linux-2.6.24-final.diff
  
  
• cisco_skbuff_offset.patch
  
  

Unpack and patch source

Next, do the following, assuming you've put all this in the same directory or sandbox:

[testbox]$ tar -zxvf vpnclient-x86_64-4.8.01.0640-k9.tar.gz
[testbox]$ mv vpnclient vpnclient-x86_64-4.8.01.0640-k9
[testbox]$ cd vpnclient-x86_64-4.8.01.0640-k9
[testbox]$ patch -p1 < ../vpnclient-linux-2.6.24-final.diff
[testbox]$ patch -p1 < ../cisco_skbuff_offset.patch

...we'll also need to do a quick hack on the 'Makefile' to make `make` happy.

[testbox]$ cp Makefile Makefile.orig
[testbox]$ sed -i -r -e "s/^CFLAGS/EXTRA_CFLAGS/g" Makefile

Compile Cisco VPN source

Ok, now the part you've been waiting for: compile time. Do the following (noting that doing the install requires r00t level access, so I use `sudo` for my needs, do whatever suits yours):

[testbox]$ cd vpnclient-x86_64-4.8.01.0640-k9
[testbox]$ sudo ./vpn_install

...and then follow the on-screen instructions for installation path, etc. etc. etc.

Testing the Cisco VPN client

Assuming that the compilation of the kernel modules went successfully, now it's time to test out the Cisco VPN client. First, make sure you're actually connected on 'some' network that's going to allow you to get to your VPN. Second, make sure you copy your Cisco VPN profile out in /etc/CiscoSystemsVPNClient/Profiles so you can actually connect to your VPN.

After that, do the following:

[testbox]$ sudo /etc/init.d/vpnclient_init start
[testbox]$ sudo vpnclient connect [name_of_profile_you_copied]

...and that's pretty much about it in a quick and dirty way. Hopefully this helps someone. I wasn't going to even attempt to start hacking code to see what was really broken. I had google'd around and noticed that people had it working; the tough part was finding all the correct patches to go with certain code bases (e.g. 64-bit for my 64-bit OS I'm running).

The re-birth of the subnotebook?

It's funny how things in our culture resurface themselves after decades of silence. From fashion to technology and anywhere in between, it's crazy the amount of odd things that are all the sudden "the thing to have" today.

Odd shit like kids wearing straight leg pants that were cool in the 80's, girls wearing their hair in a pony tail that drags off to the side of their shoulder, and most notably (of course I'm not going to bitch about fashion) the re-birth of the subnotebooks like the Asus Eee and Everex Cloudbook.

...so I ask myself: WTF happened to the Gateway Handbook? I happen to own one of these (and still do) and at the time, I used to run my own homebrew Linux distro on it (Fd Linux), in which I ripped out the 80mb 2.5" hdd and tossed in a IDE micro-controller-to-compact flash, which essentially, despite the shear lack of processing power that the Asus Eee and the Everex Cloudbook boast, isn't much different.

I'd definitely have to say, we are creatures of habit and repetition.

Start of an Era

I've had many blogs in the past; some of which were an attempt to get my thoughts and ideas out to this sick, yet interesting, digital world, others which were a spur-of-the-moment venture based solely on humiliation of others.

I've quickly come to realize that it's really difficult to categorize the multitude of passionate topics that I love to rant about, whether it's a breaking news bulletin, fantasization about starting someone on fire, black ninja conspiracies, something outlandish or retarded a fellow human being has done in my presense, or even that lovely karma that always seems to bite me in the ass daily; it's hard to manage it all.

...solution: one blog, many topics. (what a revolutional idea?)