Using newzbin with hellanzb and hellafox in Linux

I'm a huge newzbin fan! However, I could never find anything intuitive out there that was newzbin file friendly in the Linux world, as far as a .nzb file parser/reader and pull all that wonderful 'content' down from my ISP's Usenet (news) server.

I found a cool app out there called 'hellanzb'. It's a command line tool to process, queue, download or pull down .nzb files from great ease from newzbin's webstie (not to mention the automatic capability to unrar, unzip or unpack your downloads...mmm). Use it in conjunction with 'hellafox', and you've got a nice, streamlined ticket do download all the indexed Usenet 'content' a person would want.

I figured this was blog worthy, because there were some pretty shitty directions out there to get this going.

Prerequisites

Since I'm using Fedora 7 on my laptop, I had to make sure these packages were installed: python, python-twisted, unrar and par2cmdline. Again, this might not be 'everything' you need to be successful, but it's a good start for getting `hellanzb` and it's auto-unrar-ing and par2 checking features working.

If you don't have any of these installed, then just type this:

[testbox]$ yum install python python-twisted unrar par2cmdline -y

Download and Compile hellanzb

1) Download hellanzb tarball from http://www.hellanzb.com

[testbox]$ cd /home/user/sandbox
[testbox]$ wget http://www.hellanzb.com/distfiles/hellanzb-0.13.tar.gz

2) Unpack the tarball

[testbox]$ tar -zxvf hellanzb-0.13.tar.gz

3) Compile and install (as super-user)

[testbox]$ cd hellanzb-0.13
[testbox]$ sudo python setup.py install

Setting up and Configuring hellanzb.conf

The hellanzb python script (hellanzb.py) is defaultly installed in /usr/bin. As for the configuration file (hellanzb.conf), it's defaultly installed in /usr/etc. At install time, the file /usr/etc/hellanzb.conf.sample exists, so lets make a real, working copy of that so we can edit it:

[testbox]$ sudo cp /usr/etc/hellanzb.conf.sample /usr/etc/hellanzb.conf
[testbox]$ chmod 755 /usr/etc/hellanzb.conf

...now open the file with everyone's favorite text editor, let's edit hellanzb.conf and make some changes:

[testbox]$ sudo vi /usr/etc/hellanzb.conf

Here my example hellznzb.conf configuration file, with the most important settings highlighted in red:

# Log output to this file, set to None (no single quotes) for no logging
Hellanzb.LOG_FILE = '/var/tmp/hellanzb.log'

# Uncomment this line to log DEBUG messages to the specified file
Hellanzb.DEBUG_MODE = '/var/tmp/hellanzb-debug.log'

# Automatically roll over both log files when they reach LOG_FILE_MAX_BYTES
# size
Hellanzb.LOG_FILE_MAX_BYTES = 2048

# Save LOG_FILE_BACKUP_COUNT of those rolled over log files
Hellanzb.LOG_FILE_BACKUP_COUNT = 0

defineServer(id = 'servergrp',
hosts = [ 'your.news.server..hostname.net:119' ],
#hosts = [ 'news.changeme.com', 'morenews.changeme.com:8000' ],



# If your ISP allows you to log onto their news server with a login/password,
# uncomment and put those credentials in below
#username = 'usenetID',
#password = 'usenetPASS',

# If your ISP does not have credentials for you to access it's newserver, uncomment
# these lines below
username = None, # no auth
password = None,


connections = 4,
antiIdle = 4.5 * 60, # 4 minutes, 30 seconds, 0 to disable
#bindTo = '204.31.33.7', # connect FROM this ip address
#enabled = False, # disable this server
#skipGroupCmd = False, # skip sending nntp GROUP commands
#fillserver = 0, # defaults to 0 (a main server).
# fillservers must have values > 0
# (priority)
ssl = False
)

# Uncomment this line to limit all server connections to the specified KB/s
# bandwidth
#Hellanzb.MAX_RATE = 150 # limit to 150kB/s

# Important locations
Hellanzb.PREFIX_DIR = '/home/user/newzbin/'

# Where to put queued .nzb files
Hellanzb.QUEUE_DIR = Hellanzb.PREFIX_DIR + 'nzb/daemon.queue/'

# Where the fully processed archives go
Hellanzb.DEST_DIR = Hellanzb.PREFIX_DIR + 'completedownloads/'

# The .nzb currently being downloaded is stored here
Hellanzb.CURRENT_DIR = Hellanzb.PREFIX_DIR + 'nzb/daemon.current/'

# The archive currently being downloaded is stored here
Hellanzb.WORKING_DIR = Hellanzb.PREFIX_DIR + 'nzb/daemon.working/'

# Archives interrupted in the middle of downloading are stored here temporarily
Hellanzb.POSTPONED_DIR = Hellanzb.PREFIX_DIR + 'nzb/daemon.postponed/'

# Archives currently being processed. May contains archive directories, or
# symbolic links to archive directories
Hellanzb.PROCESSING_DIR = Hellanzb.PREFIX_DIR + 'nzb/daemon.processing/'

# Temp storage
Hellanzb.TEMP_DIR = Hellanzb.PREFIX_DIR + 'nzb/daemon.temp/'

# Filename to store hellanzb state in between CTRL-Cs. The state (includes the
# order of the queue, and smart par recovery information) is intermittently
# written out as XML to this file
Hellanzb.STATE_XML_FILE = Hellanzb.PREFIX_DIR + 'nzb/hellanzbState.xml'

# _Sub directory within the nzb archive dir_ to move processed files to
Hellanzb.PROCESSED_SUBDIR = 'processed'

# Remove the PROCESSED_SUBDIR if the archive was successfully post processed.
# Warning: The normal Hellanzb.LOG_FILE should be enabled with this option --
# for a record of what hellanzb deletes
Hellanzb.DELETE_PROCESSED = True

# Maximum amount of memory used to cache encoded Article data segments.
# hellanzb will write article data to disk when this cache is exceeded
# Available settings:
# -1: Unlimited size
# 0: Disable cache (only cache to disk)
# >0: Limit cache to this size, in bytes, KB, MB, e.g.:
# 1024 '1024KB' '100MB' '1GB'
#Hellanzb.CACHE_LIMIT = 0


# Save archives into a sub directory of DEST_DIR named after their newzbin.com
# category (when queued using the enqueuenewzbin XMLRPC call); e.g. Apps,
# Movies, Music
Hellanzb.CATEGORIZE_DEST = True

# Disable SMART_PAR (download all PAR files)
Hellanzb.SMART_PAR = False
#Hellanzb.SMART_PAR = True

# Supply a path to the (un)rar command
# In case in cannot find `unrar` in your PATH environment
#Hellanzb.UNRAR_CMD = None

# Supply a path to the par2 command
# In case in cannot find `par2` in your PATH environment
#Hellanzb.PAR2_CMD = None

# Skip unraring during post processing
Hellanzb.SKIP_UNRAR = False

# Supply a path to the optional macbinconv command (for converting MacBinary
# files)
#Hellanzb.MACBINCONV_CMD = None

# hellanzb inherits the umask from the current user's environment (unless it's
# running in daemon mode). The umask can be forced with this option
Hellanzb.UMASK = 0022

# Supported music types (case insensitive) and optionally their decompression
# executables
# and the file type that executable will decompress to (case insensitive). The
# exes must be in the PATH.
#
# will be replaced with the name of music file
# optional is with the specified extension
#
# None means these files don't need to be decompressed
defineMusicType('wav', None, None)
defineMusicType('mp3', None, None)
#defineMusicType('ape', 'mac -d', 'wav')
#defineMusicType('flac', 'flac -d -- ', 'wav')
#defineMusicType('shn', 'shorten -x < > ', 'wav')

# Max files we should decompress at the same time
Hellanzb.MAX_DECOMPRESSION_THREADS = 2

# Enable Mac OS X Growl notifications
Hellanzb.GROWL_NOTIFY = False

# The growl notification server, in the format 'hostname'
Hellanzb.GROWL_SERVER = 'IP'

# The growl password
Hellanzb.GROWL_PASSWORD = 'password'


# Enable libNotify Daemon notifications
Hellanzb.LIBNOTIFY_NOTIFY = False


# Disable ANSI color codes in the main screen (preserves the in place scroller)
#Hellanzb.DISABLE_COLORS = False

# Disable ALL ANSI color codes in the main screen (for terminals that don't
# support ANY ANSI codes
#Hellanzb.DISABLE_ANSI = False


# Hostname for the XMLRPC client to connect to. By default, localhost
Hellanzb.XMLRPC_SERVER = 'roboto'

# Port number the XML RPC server will listen on, and the client will connect to.
# Set to 'None' (without the quotes!) for no XML RPC server
Hellanzb.XMLRPC_PORT = 8760

# Password for the XML RPC server. You might probably never use this, but the
# command line XML RPC calls do -- it should definitely be changed from its
# default value. The XML RPC username is hardcoded as 'hellanzb' -- E.g. URL:
# http://hellanzb:changeme@localhost:8760
Hellanzb.XMLRPC_PASSWORD = 'newzbin'

# Username/Password to http://www.newzbin.com for automatic NZB downloading
Hellanzb.NEWZBIN_USERNAME = 'newzbinacct'
Hellanzb.NEWZBIN_PASSWORD = 'password123abc'

# If any of the following file types are missing from the archive and cannot be
# repaired, continue processing because they're unimportant (case insensitive)
Hellanzb.NOT_REQUIRED_FILE_TYPES = [ 'log', 'm3u', 'nfo', 'nzb', 'sfv', 'txt' ]

# Don't get rid of (move into the PROCESSED dir) the following file types when
# finished post processing (case insensitive)
#Hellanzb.KEEP_FILE_TYPES = [ 'log', 'm3u', 'nfo', 'nzb', 'sfv', 'txt' ]
Hellanzb.KEEP_FILE_TYPES = [ 'nfo', 'txt' ]


# List of alternative file extensions matched as NZB files in the QUEUE_DIR.
# The 'nzb' file extension is always matched
#Hellanzb.OTHER_NZB_FILE_TYPES = [ 'xml' ]

# Support extracting NZBs from ZIP files with this suffix (case insensitive) in
# QUEUE_DIR. Defaults to '.nzb.zip'. Set to False to disable.
#Hellanzb.NZB_ZIPS = '.nzb.zip'

# Support extracting NZBs from GZIP files with this suffix (case insensitive)
# in QUEUE_DIR. Defaults to '.nzb.gz'. Set to False to disable.
#Hellanzb.NZB_GZIPS = '.nzb.gz'

# Delay enqueueing new, recently modified NZB files added to the QUEUE_DIR until
# this many seconds have passed since the NZB's last modification time (defaults
# to 10 seconds)
#Hellanzb.NZBQUEUE_MDELAY = 10

# Optional external handler script. hellanzb will run this script after post
# processing an archive, with the following arguments:
#
# handler_script type archiveName destDir elapsedTime parMessage
#
# type: post processing result, either 'SUCCESS' or 'ERROR'
# archiveName: name of the archive, e.g. 'Usenet_Post5'
# destDir: where the archive ended up, e.g. '/ext2/usenet/Usenet_Post5'
# elapsedTime: a pretty string showing how long post processing took, e.g.
# '10m 37s'
# parMessage: optional post processing message. e.g. '(No Pars)'
Hellanzb.EXTERNAL_HANDLER_SCRIPT = '/home/user/newzbin/post_results_nzb.pl'

...most of that configuration file is self-explainatory. If you're in a hurry to get it up and go, definitely pay attention to the hosts, username, and password lines for your newsserver in the defineServer () block and if you decide NOT to use hellafox, put in your newzbin account credentials in the NEWZBIN_USERNAME and NEWZBIN_PASSWORD or you'll pretty much have no luck downloading anything or browsing newzbin indexes via command line.

One of the coolest features (I think) is the last line of the configuration file, which EXTERNAL_HANDLER_SCRIPT. This is *WAY* cool in my opinion! It basically gives you flexibility to have hellanzb put download posting result parameters into your custom script for notification when your downloads are complete!

...of course, I just had to try this out and it's 'hella' wicked (what a pun!) I whipped up a very crude perl script that will pop me up a simple perl/Tk GUI with the download results, so I'll get instant notification when my downloads are complete:

#!/usr/bin/perl
#
#
# Creation Date: 4/13/2008
#
# Script Name: post_results_nzb.pl
#
# Author: Adam Dosch
#
# Description: A post notification script written perl/Tk to notify me when my Usenet downloads are finished via hellanzb using newzbin indexing files (.nzb files)
#
# Licensing: Free to use and distribute, just give credit where it's due (e.g. sling your OWN code if you're gonna rip off mine)
#
#################################################################################################################
# START OF SCRIPT - DO NOT EDIT BELOW UNLESS YOU KNOW WHAT YOU ARE DOING
#################################################################################################################

#
# Perl Library Definitions
use Tk;

#
# Subroutines

sub quit_script
{
exit 0;
}

#
# Processing Input parameters from HellaNZB
#
# Here's the input we are expect, straight from hellanzb.conf:
#
# handler_script type archiveName destDir elapsedTime parMessage
#
# ARGV[0] -> type: post processing result, either 'SUCCESS' or 'ERROR'
# ARGV[1] -> $archiveName: name of the archive, e.g. 'Usenet_Post5'
# ARGV[2] -> $destdir: where the archive ended up, e.g. '/ext2/usenet/Usenet_Post5'
# ARGV[3] -> elapsedtime: a pretty string showing how long post processing took, e.g.
# '10m 37s'
# ARGV[4] -> parmessage: optional post processing message. e.g. '(No Pars)'
#

# Because we expect 6 parameters at all times, I just did it the crude way and
# use @ARGV without validating anything other than if we have parameters or not.
# What a programmer, eh?


if (@ARGV) {

#
# Main Window that shows post results

my $mw = new MainWindow;
my $labelheading = $mw -> Label(-text=>"NewzBin Download - $ARGV[0]") -> pack();
my $labelspacer1 = $mw -> Label(-text=>"") -> pack();
my $labelname = $mw -> Label(-text=>"Name: $ARGV[1]") -> pack();
my $labeldir = $mw -> Label(-text=>"Directory: $ARGV[2]") -> pack();
my $labeltime = $mw -> Label(-text=>"Elasped time to process: $ARGV[3]") -> pack();
my $labelpars = $mw -> Label(-text=>"Any PARS?: $ARGV[4]") -> pack();

my $button = $mw -> Button(-text => "Quit", -command => sub { exit 0} ) -> pack();
MainLoop;

} else {
# No paramters or something crazy happened!
die "No parameters passed!" && exit 1;
}

...once you've got everything configured the way you like, now it's time to use hellanzb!

Start hellanzb

To start `hellanzb`, it's quite simple. It's installed in /usr/bin/ and it's called 'hellanzb.py'. Just as long as you made your hellanzb.conf file permissions world readable (755), then you can just start up the hellanzb app either in standalone/debug or in daemon mode as yourself (non-root).

To start it up as a standalone/debug mode, which will tie up a terminal, but put all the server output to SDTOUT, do:

[testbox]$ /usr/bin/hellanzb.py

You can also daemon-ize this (which I recommend), by doing:

[testbox]$ /usr/bin/hellanzb.py -D

...if you enabled any of the hellanzb logging in your configuration file and you want see or troubleshoot something that's going on, just `tail` the log files hellanzb.log and hellanzb-debug.log which were defaultly located in /var/tmp.

Using hellanzb

I'm not going to dive into the semantics of using the command-line operations of `hellanzb` because this blog post would be HUGE and I'm not going to go into the ways I power-use it. If you want a list of all the command-line syntax and options for hellanzb, type:

[testbox]$ /usr/bin/hellanzb.py -h

However, the easiest way if you don't want to integrate hellafox into the mix (which is the hellanzb FireFox extension), is to:

1) Log onto www.newzbin.com with your newzbin credentials

2) Find the newzbin file you want to download, download it and save it to the Hellanzb.QUEUE_DIR location that you defined in your configuration file and hellanzb will automatically start processing and downloading your stuff.

3) Then if you happen to use the notification script or if you're tail'ing the log files to see when your download is done, go out to your Hellanzb.DEST_DIR and there it will be!

...so, that's the quick and dirty version to getting hellanzb working for you under Linux! If you want to configure hellafox, the FireFox browser extension, which will allow you to surf the newzbin.com website and send all your requests to download with a few mouse-clicks, continue on!

Downloading and Installing hellafox

`hellafox` was kind of tricky to get going, but I essentially went to FireFox's add-ons site and installed the add-on and restarted the browser. I do highly recommend reading hellafox's site for how to do this, so I don't have to completely reinvent the wheel.

Configuring hellafox

To configure hellafox will rely on the following configuration options you set up in your hellanzb.conf file above. Make sure you have the following information available from your configuration file, as you will need it to set up the hellafox add-on:

Hellanzb.XMLRPC_SERVER #Hostname of the machine running hellanzb, according to hellafox setup
Hellanzb.XMLRPC_PORT #The port upon which hellanzb is listening, according to hellafox setup
Hellanzb.XMLRPC_PASSWORD #The password used to connect to hellanzb, according to hellafox setup

To configure hellafox:

1) Launch Firefox

2) Goto Tools > Add-ons

3) Click the "Preferences" button under 'Hellafox firefox extension' add-on and put in the following information you gathered above (RPC server hostname, listening port and password)

Once that is done, simply do this:

1) Log onto www.newzbin.com with your newzbin credentials

2) Scour the site for whatever you're looking for, but like the hellafox website directions say, just right-click on the post link you want to download and select "Downlad this NZB"

...and away you go! Have fun and good luck!