Getting Q-logic 2532/2562 (qla2xxx) and Linux-IO SCSI Target Working On RedHat/CentOS 6.7

Having oodles of enterprise shit at work makes me a bit spoiled, so it's nice to have some sliver of that at home. So one of the things I've wanted to do for some time now is have as-real-as-possible-to-me fibre channel SAN to work with at home but do it as minimalistic as possible.

However, a few things that won't be happening:

1. Not buying and/or firing up any sort of rack-mount entry/mid/enterprise storage array at my place
2. Making my eletric bill any more hefty from the amount of shit already running in my little datacenter close
3. Have my wife draft up divorce papers for increasing the amount of space I'm already taking up with my stuff

A few things that are essential requirements (for me) for this:

1. Reusing whatever hardware I have at my place and not buying anything of significant real expense
2. Keep a small footprint (e.g. contained a mini-ATX case at most)
3. Operational and managable on top of a full OS installation of CentOS/RHEL 6.x

So, the compromise to all of this is a cheap, Brocade fibre channel switch and using Linux-IO SCSI target package.

If you did any sort of high-level googling, you'd see that are successful and documented examples of this here, here and here with newer Linux distro's using Linux Kernel 3.x and Linux-IO 3.x packages.

Reason For Doing This

The reason I am doing this:

• Because I need something to do on my 2015 holiday break from work
• I stubbornly still run RHEL/CentOS 6.x distros because of my ever loving HATE of 'system-d'
• RHEL/CentOS 6.x uses a stable but relic 2.6.x kernel --- which Q-Logic HBA fibre channel target mode is not merged into (obviously)
• Even with a support Linux kernel that has Q-Logic HBA fibre channel support (3.x+), the Linux-IO packages for RHEL/CentOS 6.x (fcoe-target-utils) still won't work and you have to re-build those so it picks up the support in configFS under sysfs.  Since the tools are written in python, without digging into any code in 'targetcli' or 'rtslib', I'm leaning towards 'rtslib' being the outdated piece of fcoe-target-utils.
• I'm sure one could have tracked and gathered a bunch of patches to see if it's possible to merge the tcm_qla2xxx + sysfs changes into the stock CentOS 6.7 (2.6.32-573.el6.x86_64) but it's just easier to re-compile the whole thing.

What Else this is useful for

There seems to be a lot of buzz off other people's blog posts about even folks using RHEL/CentOS 7.x not being able to get qla2xxx working.  Again, I lean on either parsing in 'targetcli' or 'rtslib' as a complete wild ass guess being the issue, since tcm_qla2xxx module is already pre-compiled and availbale with the stock kernels.

My advice would be just skip down to the latest-and-greatest Linux-IO compilation builds to get yourself going.

Install Needed Packages

Depending on how you baseline your RHEL/CentoOS 6.x installs, you'll definitely want to install these packages via 'yum' (and possibly more).  I'll order these by the groups of things you'll need the packages for just so it makes sense as far as what is depended for what.

For Linux kernel compiling:

$ yum install gcc ncurses-devel kernel-devel lsscsi patch git -y

For Linux-IO configshell, rtslib and targetcli latest rebuild:

$ yum install python-devel epydoc python-pyparsing python-configobj python-prettytable -y

Extra RPMs you will have to hand-install and can easily find on rpm.pbone.net for Linux-IO rebuild:

• python-pyparsing
• python-netifaces

Build Linux 3.x Kernel (3.18.25)

This is a pretty clean-cut set of step(s) to get the kernel rebuilt and do the bare minimum tweak of enabling the build of the Q-logic TCM_QLA2XXX fabric module for Qlogic 2xxx series target mode HBAs:

Download and Unpack Linux Kernel Source

First step is to download and unpack the Linux kernel source:

$ wget --no-check-certificate https://cdn.kernel.org/pub/linux/kernel/v3.x/linux-3.18.25.tar.xz
$ tar zxvf linux-3.18.25.tar.xz

Copy Existing Kernel Configuration and Configure Kernel Build

Next is to copy over our latest stock kernel configuration for our CentOS as a starting configuration point so, if you want, you can build at a minimum what you have configured in your big, bloated distro kernel plus whatever you are going to enable.  Do note, there are numerous changes between 2.6 and any 3.x or 4.x kernel, so again, I'm not advocating this, but for the sake of 'getting shit done via blog directions', it'll do.

Let's change directory into the kernel source and run our ncurses text-based kernel configuration menu:

$ \cp /boot/config-2.6.32-573.el6.x86_64 /path/to/linux-3.18.25/.config
$ cd linux-3.18.25
$ make menuconfig

You'll want to, at a minimum, navigate to the following menu tree to enable Q-logic TCM_QLA2XXX fabric module for Qlogic 2xxx series target mode HBAs:

Location:
   -> Device Drivers                        
     -> SCSI device support           
       -> SCSI low-level drivers (SCSI_LOWLEVEL [=y])
         -> QLogic QLA2XXX Fibre Channel Support (SCSI_QLA_FC [=m])
          -> [m] TCM_QLA2XXX fabric module for Qlogic 2xxx series target mode HBAs

You can toggle on module (e.g. m) build by using your space-bar.  Then exit out of the menu's and save the configuration.

If you're super leery if you did the right thing or not, run this:

$ cat linux-3.18.25/.config | grep -q "CONFIG_TCM_QLA2XXX=m" && echo "enabled" || echo "not enabled"
enabled
$

...if you see 'not enabled' echo'd on the screen, you didn't get it enabled.  So you can either hand-edit that configuration line into '.config' or go back through ncurses menuconfig and re-enable it.  Either or will work.

Once your configuration is saved, we can build the linux kernel.

Build the Kernel and Modules

To build the kernel and it's accompanied modules, simply type:

$ make -j4 bzImage modules

...then go get some coffee.  If you compile with all the CentOS configured extra modules and add-in's you had with your stock os kernel, you're looking at just under one hour for a multi-threading build above.

Also, you'll definitely want to adjust '-j <n>' to your tastes; this runs automake with <n> threads.  Try to, at a minimum, make <n> equal to the number of cores you have on your system you are building on.

Install and Boot Into 3.x Kernel

After your kernel and modules build (successfully, I hope), it's time to install the kernel and the modules onto your system.  The linux kernel build strap scripts have come a LONG ways over the years and they work pretty flawelessly.

Add 'qla2xxx' target mode parameter for kernel module

This is an important step so prior to using the kernel build strap scripts for installing the kernel modules and invoking 'dracut', will automagically grab these options and incorporate them into your ramdisk (e.g. initrd).

Run the following command(s) to add the target mode enable options for the qla2xx fibre channel module:

$ echo "options qla2xxx qlini_mode=\"disabled\"" >> /etc/modprobe.d/qla2xxx.conf

Install Kernel, Modules and config

To install the kernel modules, do the following:

$ make modules_install

To install the kernel, do the following:

$ make install

This isn't a necessary step, but for the sake of making our custom kernel 'look and feel' the same, let's copy over the configuration as well to the /boot directory:

$ cp linux-3.18.25/.config /boot/config-3.18.25

Double-check the GRUB2 entries in '/etc/grub.conf' as well.  If you're super paranoid, you may also want to look at the contents of your in ramdisk (e.g. initrd) to make sure you got the 'qla2xxx' and 'tcm_qla2xxx' modules in there.

Reboot into New Kernel

Let's reboot your system:

$ init 6

We are going to assume you're going to leave your existing, distro-built kernel as 'default' and you'll manually select the 3.18.25 kernel from your GRUB menu.

If you don't and want to boot into this from the get-go without any intervention (e.g. maybe you're running a headless box like I am OR you're cocky as shit), just edit '/etc/grub.conf' and change the following line back to:

default=0

...so the bootloader defaults to the first-most kernel entry, which the linux kernel build strap scripts will put your new kernel at the top of the available kernels-to-boot list.

Do whatever you want to do.

Verify Kernel Installation

To do a quick kernel installation verification, obviously having the system back and booted is a probably the top indicator.

Next, let's verify the kernel we are booted into is, in fact, the one we built (sounds funny, but I have shot myself in the foot MANY times by overlooking that and skipping a few steps doing this by-the-cuff):

$ uname -r
3.18.25
$

Next, lets verify that our Q-logic module(s) can be loaded (assuming the card isn't plugged into any SAN fabric or have any existing storage attached/zoned to it).  You should see similar when running 'dmesg':

$ dmesg | grep -i qla
qla2xxx [0000:00:00.0]-0005: : QLogic Fibre Channel HBA Driver: 8.07.00.16-k.
qla2xxx [0000:02:00.0]-001d: : Found an ISP2532 irq 16 iobase 0xffffc90004afe000.
qla2xxx 0000:02:00.0: irq 26 for MSI/MSI-X
qla2xxx 0000:02:00.0: irq 27 for MSI/MSI-X
scsi host10: qla2xxx
qla2xxx [0000:02:00.0]-00fb:10: QLogic QLE2560 - PCI-Express Single Channel 8Gb Fibre Channel HBA.
qla2xxx [0000:02:00.0]-00fc:10: ISP2532: PCIe (5.0GT/s x8) @ 0000:02:00.0 hdma+ host#=10 fw=7.03.00 (90d5).
$

Lastly, lets load the target SCSI module we added and make sure that loads:

$ modprobe tcm_qla2xxx

Build the Latest Linux-IO Packages

Below are to-the-points steps to rebuild the latest and greatest Linux-IO packages you'll need since even after having the new kernel that supports Q-logic 256x target support, the current relic 'targetcli' package set still won't work.

Make sure 'fcoe-target-utils is Uninstalled

Before we go any further, let's make sure anything related to the stock distro 'fcoe-target-utils' is uninstalled and NOT on our system:

$ yum remove fcoe-target-utils python-rtslib python-configshell -y

Build and Install Configshell

To build and install the latest/greatest configshell, do the following command(s):

$ git clone https://github.com/Datera/configshell.git
$ cd configshell
$ make rpm
$ cd dist
$ rpm -ivh python-configshell-1.6.1~g020d540-1.el6.noarch.rpm

Build and Install RTSLib

To build and install the latest/greatest RTSLib, do the following command(s):

$ git clone https://github.com/Datera/rtslib.git
$ cd configshell
$ make rpm
$ cd dist
$ rpm -ivh python-rtslib-3.0.pre4.9~g6fd0bbf-1.el6.noarch.rpm

Build and Install TargetCLI

To build and install the latest/greatest TargetCLI, do the following command(s):

$ git clone https://github.com/Datera/targetcli.git
$ cd configshell
$ make rpm
$ cd dist
$ rpm -ivh targetcli-3.0.pre4.5~ga125182-1.el6.noarch.rpm

Verify 'targetcli' is working

Lastly, a simple verification and what will look different from using the 'fcoe-target-util' package targetcli command is the actual qla2xxx target config being loaded so you can set up your block storage backstores for export.  You'll also notice in the output of targetcli it references the rtslib build version we just build and installed via RPM, too:

$ targetcli
targetcli 3.0.pre4.5~ga125182 (rtslib 3.0.pre4.9~g6fd0bbf)
Copyright (c) 2011-2014 by Datera, Inc.
All rights reserved.

/> qla2xxx/ info
Fabric module name: qla2xxx
ConfigFS path: /sys/kernel/config/target/qla2xxx
Allowed WWNs list (free type): 21:00:00:24:xx:xx:xx:xx
Fabric module specfile: /var/target/fabric/qla2xxx.spec
Fabric module features: acls
Corresponding kernel module: tcm_qla2xxx
/> ls
o- / ............................................................................................................. [...]
  o- backstores .................................................................................................. [...]
  | o- fileio ....................................................................................... [0 Storage Object]
  | o- iblock ....................................................................................... [0 Storage Object]
  | o- pscsi ........................................................................................ [0 Storage Object]
  | o- rd_mcp ....................................................................................... [0 Storage Object]
  o- loopback .............................................................................................. [0 Targets]
  o- qla2xxx ............................................................................................... [0 Targets]
/>

That's it (in a big nutshell)! You should now be able to continue on setting up your block storage to front as a backstore from your newly set up fibre channel target.