Category Archives: Development

Converting Your Corporate Intranet to Drupal

Though I have fun working on SynthNet and other projects at night, during the day I fill the role of mild-mannered network administrator at the Manchester-Boston Regional Airport (actually, the day job is quite a bit of fun as well). One of the ongoing projects I’ve taken on is adding all of our various Intranet-oriented services into a single platform for central management, easier use, and cost effectiveness. As mentioned in a previous article (linked to below, see NMS Integration), I knew Drupal was the right candidate for the job, simply due to the sheer number of modules available for a wide array of functionality, paired with constant patching and updates from the open source community.  We needed a versatile, sustainable solution that was completely customizable but wasn’t going to break the bank.

The Mission

The goal of our Drupal Intranet site was to provide the following functionality:

  1. PDF Document Management System
    1. Categorization, customized security, OCR
    2. Desktop integrated uploads
    3. Integration with asset management system
  2. Asset Management System
    1. Inventory database
    2. Barcode tracking
    3. Integration with our NMS (Zenoss)
    4. Integration with Document Management System (connect item with procurement documents such as invoices and purchase orders)
    5. Automated scanning/entry of values for computer-type assets (CPU/Memory/HD Size/MAC Address/etc)
    6. Physical network information (For network devices, switch and port device is connected to)
    7. For network switches, automated configuration backups
  3. Article Knowledgebase (categorization, customized security)
  4. Help Desk (ticketing, email integration, due dates, ownership, etc)
  5. Public Address System integration (Allow listening to PA System)
  6. Active Directory Integration (Users, groups, and security controlled from Windows AD)
  7. Other non-exciting generic databases (phone directories, etc)

Implementation

Amazingly enough, the core abilities of Drupal covered the vast majority of the required functionality out of the box.  By making use of custom content types with CCK fields, Taxonomy, Views, and Panels, the typical database functionality (entry, summary table listings, sorting, searching, filtering, etc) of the above items was reproduced easily.  However, specialized modules and custom coding was necessary for the following parts:

  1. Customized Security – Security was achieved for the most part via Taxonomy Access Control and Content Access.  TAC allowed us to control access to content based on user roles and categorization of said content (e.g. a user who was a member of the “executive staff” role would have access to documents with a specific taxonomy field set to “sensitive information”, whereas other users would not).  Additionally, Content Access allows you to further refine access down to the specific node level, so each document can have individual security assigned to it.
  2. OCR – This was the one of the few areas we chose to delve into a commercial product.  While there are some open source solutions out there, some of the commercial engines are still considerably more accurate, including the one we choose, ABBYY.  They make a Linux version of the software that can be driven via the shell.  With a little custom coding, we have the ABBYY software running on each PDF upload, turning it into an indexed PDF.  A preview of the document is shown in flash format by first creating a swf version (using pdf2swf), then using FlexPaper/SWF Tools.
  3. Linking Documents – This was performed with node references and the Node Reference Explorer module, allowing a user friendly popup dialogs to choose the content to link to.
  4. Desktop Integration – Instead of going through the full steps of creating a new node each time, choosing a file to upload, filling in fields, etc, we wanted the user to be able to right click a PDF file on their desktop, and select “Send To -> Document Archive” from Windows.  For this, we did end up doing a custom .NET application that established an HTTP connection to the Drupal site and POSTed the files to it.  Design of this application is an article in itself (maybe soon!).
  5. Barcoding – This was the last place we used a commercial product simply due to the close integration with our barcode printers (Zebra) – we wanted to stick with the ZebraDesigner product.  However, one of the options in the product is to accept the ID of the barcode from an outside source (text/xml/etc), so this was simply a matter of having Drupal put the appropriate ID of the current hardware item into a file and automating ZebraDesigner to open and print it.
  6. NMS (Zenoss) Integration – The article of how we accomplished this can be found here.
  7. Automated Switch Configuration Backups and Network Tracking – This just took a little custom coding and was not as difficult as it might seem.  Once all our network switches were entered into the asset management system and we had each IP address, during the Drupal cron hook, we had the module cURL the config via the web interface of the switch by feeding it a SHOW STARTUP-CONFIG command (e.g. http://IP/level/15/exec/-/show/startup-config/CR) – which was saved and attached to the node.  Additionally, we grabbed the MAC database off each switch (SHOW MAC-ADDRESS-TABLE), and parsed that, comparing the MAC addresses on each asset to each switch port, and recording the switch/port location into each asset.  We could now see where each device on the network was connected.  A more detailed description of the exact process used for this may also be a future article.
  8. Help Desk – While this could have been accomplished with a custom content type and views, we chose to make use of the Support Ticketing Module, as it had some added benefits (graphs, email integration, etc)
  9. Public Address System – Our PA system can generate ICECast streams of its audio.  We picked these up using the FFMp3 flash MP3 Live Stream Player.
  10. Automated Gathering of Hardware Info – For this, we made use of a free product called WinAudit loaded into the AD login scripts.  WinAudit will take a full accounting of pretty much everything on a computer (hardware, software, licenses, etc) and dump them to a csv/xml file.  We have all our AD machines taking audit during logins, then dumping these files to a central location for Drupal to update the asset database during the cronjob.
  11. Active Directory Integration – The first step was to ensure the apache server itself was a domain member, which we accomplished through the standard samba/winbind configurations.  We then setup the PAM Authentication module which allowed the Drupal login to make use of the PHP PAM package, which ultimately allows it to use standard Linux PAM authentication – which once integrated into AD, includes all AD accounts/groups.  A little custom coding was also done to ensure matching Drupal roles were created for each AD group a user was a part of – allowing us to control access with Drupal (see #1 above) via AD groups.

There was a liberal dose of code within a custom module to glue some of the pieces together in a clean fashion, but overall the system works really smoothly, even with heavy use.  And the best part is, it consists of mainly free software, which is awesome considering how much we would have paid had we gone completely commercial for everything.

Please feel free to shoot me any specific questions about functionality if you have them – there were a number of details I didn’t want to bog the article down with, but I’d be happy to share my experiences.

BerryCraft Source – Jave Minecraft I/O Engine

Below is the source to BerryCraft.  Most of the project is pretty trivial – however, MCIO.java (MineCraft IO) contains a class capable of full communication with a Minecraft server, correctly sending/receiving all Minecraft packet types as documented here.  BerryCraft itself only implements chat/time functionality (and logins of course), but the MCIO could be used to build out any Minecraft functionality (player positioning, mob spawning and attributes, inventory, etc).

If you do end up using MCIO, let me know, I’d love to see the results!

BerryCraft Source

 

 

BerryCraft Available for Download

I fixed up the last of the bugs, and added in a small amount of new functionality, and the first iteration of the BerryCraft, the Blackberry Minecraft chat client, is now complete.

 

The first iteration includes the following functionality:

  1. Connects to Minecraft server from your Blackberry (handles both authenticated or offline modes)
  2. Allows chatting with players online (and sending commands such as /time, /list, etc if available to your account)
  3. Has 3 user-assignable macros for repeatedly used messages/commands (good for admin commands)
  4. Allows user to set launcher and protocol version, which assuming no major protocol changes, should allow the client to continue to work with future server versions.
  5. Works with BB devices OS 5.0 and up over any connection type (Wifi, BIS, MDS, etc)

Additionally, the source code is available here – it includes an IO engine that fully sends/receives all packet types and a shell Game class to implement future functionality past chatting.

Download OTA here!

BerryCraft – Update 5/13/12

I was hoping to release the first iteration of the BerryCraft client tonight, and though I got a lot done this weekend, there are still a couple errors that are preventing it from going primetime.

Most importantly, I fleshed out 99% of the Minecraft protocol IO functions, so it can send and retrieve all 68 or so packet types as documented.  Obviously it only actually implements the basic connection/chat/time functionality (as opposed to drawing anything), but as far as communicating with the server, it understands everything (just  a couple errros still popping up from things not working right).  This is almost a necessity, as the server could potentially send any packet type over, and if the client doesn’t receive it properly, it will crash – so it has to be ready for anything in the protocol specification.  Internally, it can tell when mobs are spawned/move/look, when painting/items spawn, player abilities/movements, sound effects happen, etc etc.

Also – the client stays up for as long as desired (I ran it for 30-40 minutes at one point) – so it should be stable once the fixes are done.

Hopefully it won’t take more more than a couple nights to fix the bugs – expect another post when it’s good to go!

Authenticated Minecraft Logins Working for BerryCraft

A quick update – a lot of people had shown interest in BerryCraft – the Blackberry Minecraft admin/chat client I’d been working on.  It was a bit of a hack before and didn’t work with servers that required Minecraft.net authentication, so I wanted to fix it up before starting to release it out.   Tonight I successfully got the authenticated logins working.  I want to clean it up a bit more, and then stat releasing it in iterations.  The first will be mainly just a chat client, since that’s pretty much implemented.  Then we’ll see about inventory management and some other goodies if things work out well.

Hardware Monitoring: Syncing Drupal with Zenoss

Overview

One of the more daunting tasks of managing a larger network is keeping track of all your devices – both physically, and from a network monitoring perspective.  When I arrived on the job 3 years ago, the first major task I laid down for myself was implementing both an asset management system, as well as a network monitoring system, to ensure we always knew what we had, and if it was functioning properly.

I decided almost immediately that Drupal was the right candidate for the job of asset management.  There are a number of commercial IT/helpdesk systems out there which work great, but they are usually fairly expensive with recurring licensing costs, and my history with them has always been shaky.  Plus, I find myself not always using all the functionality I paid for.  I knew with my Drupal experience, I could get something comparable up in almost no time – this is not a discredit to IT packages, but moreso the power of the Drupal framework.

Network Monitoring – Cue Zenoss

Now that I had the hardware DB taken care of, I needed a NMS for monitoring.  Originally I was planning on Nagios, but a contractor who works for us (now friend) had introduced me to Zenoss, another open source alternative.  Zenoss is awesome – is absolutely has its quirks, and is not the most intuitive system to learn, but once things are up and running it’s great – and tremendously powerful.  So the choice was made.

Now – I had both pieces, but I absolutely hate entering data twice, and the interoperability guy in me loves integrating systems.  So I decided to write a script that would sync our Drupal database with Zenoss.  Drupal would serve as our master system, and any hardware we entered into it would automatically port over to Zenoss.  Any changes or deletions we made (IP address, location, name, etc) would sync over as well.

The below script performs this synchronization.  Some warnings up front – I’m not a Python guy by any means, I specifically learned it for this script, so I apologize for any slopping coding or obvious Python-y mistakes.  I’ve tried to thoroughly comment it to document how to use it and how it works.  Hopefully it can help some others out as well!

# Description: Sync devices to be monitored from Drupal to Zenoss
#
# Setup Work: Create a (or use an existing) content type that houses your hardware items to be monitored.
# They should have CCK fields for the IP address of the device, the name, and the type of
# device it is. The device type will determine the Zenoss class the script adds it to, and hence
# the kind of monitoring it will receive (e.g. Linux server, switch, ping only, etc)
#
# Additionally, in Zenoss, create a custom property field that will house the nid of the Drupal
# node. This serves as the foreign key and will be used to link the item in Drupal to its entry in Zenoss
#
# Usage: This script should be run from zendmd, and may be run once or periodically. We run it every 20 minutes from
# a cron job.
# It will create new entries in Zenoss for items not yet imported, delete ones that no longer exist in
# Drupal (it will only delete ones that were originally imported from Drupal), and will update ones that have
# been updated (type, IP, location, etc).
#
# Note: Excuse all the extra commits - we experienced some issues with data not being saved, and I threw some extra in
# there - they're almost definitely not necessaryimport MySQLdb

# Take a taxonomy term from Drupal identifying the type of monitoring to be done,
# and convert it to the appropriate Zenoss class path. Update these to whatever terms
# and Zenoss class paths that make sense for your environment. We setup ones for
# Linux and Windows servers, switches, waps, UPSes, PDUes, etc, as can be seen.
def getClassPath(passType):

if passType.lower() == "windows":
return "/Server/Windows"
elif passType.lower() == "linux":
return "/Server/Linux"
elif passType.lower() == "switch":
return "/Network/Switch"
elif passType.lower() == "mwap":
return "/Network/WAP/Managed"
elif passType.lower() == "uwap":
return "/Network/WAP/Unmanaged"
elif passType.lower() == "ups":
return "/Power/UPS"
elif passType.lower() == "pdu":
return "/Power/PDU"
elif passType.lower() == "camera":
return "/Camera"
elif passType.lower() == "cphone":
return "/Network/Telephone/Crash"
elif passType.lower() == "sphone":
return "/Network/Telephone/Standard"
elif passType.lower() == "printer":
return "/Printer"
elif passType.lower() == "converter":
return "/Network/Converter"
elif passType.lower() == "ping":
return "/Ping"
return "/Ping"

# Connect to Drupal's MySQL DB (Replace these values with the appropriate ones for your system)
imsConn = MySQLdb.connect(DRUPAL_MYSQL_SERVER, MYSQL_USER, MYSQL_PASSWORD, MYSQL_DB)
imsCursor = imsConn.cursor()

# Execute the query to grab all your items to be monitored. In our case, we have a node type called "hardware" that had CCK fields identifying the IP address,
# the type of hardware (a taxonomy term that dictated the Zenoss class of the item - see getClassPath above), a physical location, etc.
# You'll want to change the specific table/field names, but the inner join will probably stay, as you'll want to grab both the node and CCK fields that belong to it.
imsCursor.execute("""
SELECT node.nid, content_type_hardware.field_hardware_dns_value, content_type_hardware.field_hardware_location_value, content_type_hardware.field_hardware_ip_value, content_type_hardware.field_hardware_monitor_type_value, content_type_hardware.field_hardware_switchname_value, content_type_hardware.field_hardware_switchport_value
FROM node
INNER JOIN content_type_hardware ON node.nid = content_type_hardware.nid
""")

# Loop through all returned records - Check for additions, changes, and removals
while (1):
#tempRow is our current hardware item record
tempRow = imsCursor.fetchone()
if tempRow == None:
# No more entries, break out of loop
break
else:
# Search Zenoss records for the nid of the hardware item. A custom field will need to be created in Zenoss to serve
# as this foreign key. In our case, we used MHTIMSID - but you can use anything you'd like - just be sure to create the field in Zenoss.
found = False
for d in dmd.Devices.getSubDevices():
if d.cMHTIMSID != "":
if int(d.cMHTIMSID) == tempRow[0]:
found = True
break

if found == False:
# Hardware item not found, add it if it's monitored
if tempRow[4] != None:
dmd.DeviceLoader.loadDevice(("%s.yourdomain.com" % tempRow[1]).lower(), getClassPath(tempRow[4]),
"", "", # tag="", serialNumber="",
"", "", "", # zSnmpCommunity="", zSnmpPort=161, zSnmpVer=None,
"", 1000, "%s (%s - %s)" % (tempRow[2], tempRow[5], tempRow[6]), # rackSlot=0, productionState=1000, comments="",
"", "", # hwManufacturer="", hwProductName="" (neither or both),
"", "", # osManufacturer="", osProductName="" (neither or both).
"", "", "", #locationPath="",groupPaths=[],systemPaths=[],
"localhost", # performanceMonitor="localhost',
"none")
tempDevice = find(("%s.yourdomain.com" % tempRow[1]).lower())
tempDevice.setManageIp(tempRow[3])
commit()
# Save nid to Zenoss record (to serve as foreign key) for syncing
tempDevice._setProperty("cMHTIMSID","MHTIMS ID","string")
tempDevice.cMHTIMSID = tempRow[0];
commit()
else:
# Hardware item found - delete, update, or do nothing
if tempRow[4] == None:
# Delete if not set to monitor
dmd.Devices.removeDevices(d.id)
else:
# Update DNS and IP to current values
if d.getDeviceName() != ("%s.yourdomain.com" % tempRow[1]).lower():
d.renameDevice(("%s.yourdomain.com" % tempRow[1]).lower())
if d.getManageIp() != tempRow[3]:
d.setManageIp(tempRow[3])
commit()

# Change class if not set to "Manual" (We setup a taxonomy term called "Manual" that would turn off automatic Zenoss class selection during syncing
# and allow us to manually specificy the class of the device.
if tempRow[4] != "Manual":
d.changeDeviceClass(getClassPath(tempRow[4]))
commit()

# Update comments (location change)
d.comments = "%s (%s - %s)" % (tempRow[2], tempRow[5], tempRow[6])
commit()

# Save any missed changes
commit()

# Close connection to database
imsCursor.close()
imsConn.close()

Blackberry Minecraft Chat Client

First off, if you haven’t tried Minecraft yet, it is a ridiculously addictive game in which you, the player, dig tunnels, collect materials, craft items, and build up the world around you. You explore underground caverns, build houses, castles, farms, etc. It’s a ton of fun (and definitely a good way to waste hours of your life). If you haven’t seen it before, check it out.

I run a Minecraft server for a few friends of mine, and even if I wasn’t in the game, I wanted to be able to chat with them and run server comands on the go. So I wrote a VERY quick and extremely dirty (and buggy) Minecraft chat client for the Blackberry that will connect to a server, and let you chat and run commands.

BerryCraft Chat:

It has a few limitations – specifically that it will only connect to a server that has minecraft.net authentication turned off. It wouldn’t be too difficult to insert this functionality into it, but I banged this out in a couple days and don’t really have time to put any polish on it.

UPDATE: BerryCraft can be downloaded here.

Spotlight: Leah Creates

One of the best things about being involved in the world of technology, besides getting a front row seat to all the amazing advancements made every day, is meeting and talking with the creative people who make the magic happen. I think I’m especially lucky, having strong ties to a range of different areas such as networking and development, to have met a diverse mix of very talented people.

Web Developer Extraordinaire

To say business exists in a social media world where online presence and reputation is important would be the understatement of the century. Companies today live and die by their ability to harness the power of the web. And while there are many developers out there, a true burden lies in finding talented and experienced ones. Not only does Leah fall into this camp, combining expert design skill with seasoned web development knowledge, but she possesses something that many in the industry don’t – a real love and respect for what her customers are trying to accomplish with their website. This truly shines through both in her work, and how she treats her clients. It translates into a special website that speaks its goals and connects to its visitors like no other site could. It is the difference between a good looking site and a truly powerful site.

The Proof is in the Pudding

I’ve known Leah for a number of years, having had the privilege of working with her on a number of projects professionally – and her sites continue to really impress me. Some excellent examples of recent projects: Be Irreplaceable | Donna Heart.

I love these examples, as they show how she has taken a general framework like WordPress, and turned it into a beautiful site that really communicates the site’s message. They feel personable and comfortable when you visit them, unlike a lot of cold and bland sites out there. They have that truly personal touch which is key to connecting with the audience.

For even more examples of her work, check out her online portfolio.

So if you’re looking to build a new website for your business, or need to re-imagine the one you already have, I really suggest keeping Leah Creates in mind. She is amazing at both what she does, and how she does it – something setting her apart from so many other development houses out there.

LeahCreates

Good First Week for Galactic Blast!

I tweeted this as well, but I just wanted to thank everyone who has been supportive of both the Blackberry Game Development Tutorial, as well as our commercial release of Galactic Blast! – over the first week, we sold over 100 copies! It’s a great feeling to know people are out there enjoying your game.

Thanks again everyone!

Galactic Blast Released on App World!

I’m very excited to say Galactic Blast is now officially available for purchase from App World!!

Pick it up now and save the galaxy, right from your Blackberry!

And for all the aspiring developers out there, don’t forget to check out the tutorial to make games just like this one.