Results tagged “software”

If you do any kind of application programming, particular network oriented applications or web applications (who isn’t?), you should definitely be aware of OWASP. They keep a list of the Top Ten security concerns when developing software and I have found the reference to be excellent.

• Site Link: http://www.owasp.org/index.php/Category:OWASPTopTen_Project
• PDF Link: http://owasptop10.googlecode.com/files/OWASP%20Top%2010%20-%202010.pdf

I think a printed version of the updated Top Ten is going to become a part of my standard reference kit to look over from time to time. The guide is organized and color coded in such away that just about anyone could understand why you should be concerned and what the remedies for each of the top ten issues. It provides additional links off to further information and even a standard reference library that provides an implementation reference for best practices.

Cheers.

This morning I had an epiphany about a difference in project management style between the two major development jobs I’ve held. One style was like driving an empty bus and the other was like riding in a clown car. I am going to examine both as anecdotes from my perspective and try to avoid grandiose analysis.

The Empty Bus

So, I start the job and the first thing the company does is hand me the keys to the bus. Actually, they dither on what kind of bus to give me for several months before getting me a suitable one and give me a loaner to drive in the meantime. However, once on the bus driving, I am pretty much on my own. I have a destination to reach that has been vaguely described on a scribbled piece of paper. The directions are unclear and no one in the company has been there before. They keep changing the directions. But I get to drive. That is fun.

Every now and then, I pick somebody from the company up, they make changes to the directions and then they get off again before I go very far. Every six months, everybody climbs on to the bus and sits in the very back. They do a lot of yelling while I park and then they take away my scribbled directions and give me new scribbles to follow and a new destination to reach. But I get to drive. That’s usually fun.

All in all, I am asked to develop software with very little cooperation or direction. I am left on my own to make almost all the decisions. Even though I have weekly meetings with my manager, I am not really given much feedback on whether I’m going the right way. He’s not a developer, so he doesn’t really know enough about what I do to give me useful feedback. My quarterly reviews aren’t very cooperative or helpful, they are more about the manager wishing I would drive faster and make fewer mistakes (mutually exclusive goals when you think about it).

I nearly get into a wreck a couple times, but there’s no one on the bus to help me out. Don’t get me wrong, I’m not an excellent driver. I’m still learning, but some help should help things go faster in process, you’d think. Usually, though, my directions are so unclear and difficult to follow that I am directed to get into wrecks. This is not actually all that fun as time goes on.

The Clown Car

Clown cars are funny. They drive around in circles and then the doors pop open and an absurd number of hilarious characters hop out of the little car. This job is not quite like this. It’s actually more like a really crowded minivan, like the trip I took the other day with my wife, son, dad, mom, brother, sister, and her boyfriend, all crammed into our little Pontiac Montana. But now, imagine, that all of these people have a stake in where the van goes and have a slightly role and different idea about how to go about getting there. Now, we’ve got a good analogy. Clown cars are fun, though.

The CEO’s seat is next to mine and he gets in and out of the van whenever he feels the need. He’s a busy man: lots of vans to help. Usually, he gets in right before we wreck or near the major turns to make sure we drive carefully at those point and turn the right direction. Directly behind me sits an analyst whose job it is to navigate. He tells me where to go and annotates those instructions pretty regularly. Beside him sits another analyst whose job it is to talk to the customer and figure out what they want. He then tells the first analyst and me where we need to think about going next. I get to drive, though, they keep reaching from the back for the wheel and the pedals. That’s annoying, but still fun.

Behind the analysts sit a whole team of project managers, executives, sales people and between them and between the front seats sit a bunch of other engineers. Sometimes the other engineers help drive, make suggestions, and they often critique or commend my driving. There are a lot of people in this minivan, sometimes there’s a lot of yelling about what to do next. All the activity does make for quite a bit of fun, even if it gets a bit distracting at times.

I get to drive. As I mentioned, sometimes the analysts and engineers have their hand on the wheel and help push the pedals for me. This is pretty fun too, unlike the car though, this actually gives us a lot more control. We seem to be getting places in a much more controlled way, though we do have to control our speed much more carefully. It might take us longer, but the drive is fun along the way.

Software development in this style takes away some of my freedom as a coder. That’s a bummer in some ways, not as much fun. I like control. Yet, it also lets me focus on my strengths while others worry about talking to customers, making sure we have a plan that does what the customer wants, and while the constant feedback makes it hard to see the big picture, I usually have a lot of warning before I drive off the road or get into a wreck. Overall, this is more fun.

So far, I prefer the “clown car”/stuffed minivan to the empty bus. It’s less bipolar and more slow, steady, and directed.

Cheers.

Sorry for the Sir Mixalot reference there. Ahem. Anyway, I just had a coworker walk by my desk and say something like, “Don’t you just wish we could get rid of all these glitches and be done with them?” She left before I could answer and I think it was rhetorical (it’s hard for me to switch gears to interpret social signals while in the midst of concentrating on code). My answer was going to, “Uh, no. I like finding bugs.” In this sense, I’d agree with what Nat Torkington twittered recently (edited for language, I try to run a PG rated blog):

acid test of whether I’m still a hacker: do I think “oh goody!” or “oh [skittles]!” when I find a bug?

Not only this, but a few years back I worked for a company, NRG, that shared office space with another company owned by John Devore. John and I were chatting one afternoon and we were talking about debugging things. He said that when he finds a bug, he doesn’t just want to fix it. He wants to know why he didn’t find it before: why does the bug work so well except here? That’s certainly my passion.

Debugging is an interesting opportunity to learn. Not only can you learn from the mistake, but you can then use that mistake to actually become an enhancement in the future if it happens to be interesting in some way. Sometimes, you can even end up turning a bug into a feature if it happens to do something really cool with a slight change.

Anyway, I just wanted to post that I enjoy debugging and when I get to the point where I say, “Ah crap” when I need to fix something, it’s time to switch careers.

Cheers.

I have a certain frustration with due dates. I have determined that this frustration stems from a misunderstanding of the nature of software and their timelines. I am, therefore, banishing the terms "due date" and "deadline" from the realm of software project timelines (as a lot of software engineering literature has wisely already done) and insist that the word "milestone" be used in its place. Why?

When you decide to create a piece of software to perform some task, you are starting down a path that has no end. Once a software project is created, that project immediately takes on an eternal soul. If you want to add feature X to the software by deadline Y, you will soon find that even if you succeed in getting this feature in by the deadline you still have more work to do on that feature after the deadline. Why?

Features are nebulous little critters. Even when you think you have them well defined, you find out they are more complex once actually implemented. For example, I'm in the process of converting our server infrastructure to use a tool that will help us scale upwards as we add more clients. We aren't done once we move our servers to this new infrastructure because we'll still need to retool it and play with the optimization settings to make sure it's performing well. This will continue for as long as we have the servers.

There's no finality on a software project ever. It will always require a little more maintenance until the business decides we're not doing that anymore. Even then, depending on the project, we may need to do some level of support down the road. I would be surprised if Microsoft doesn't still get support calls for DOS or Windows 95, which have been dead products for years.

My conclusion: A software project is never finished. A deadline is not a deadline, it's a milestone. "Deadline" and "due date" imply finality. A "milestone" is just another landmark along the path, a much better description. There's always one more bug to knock out. There's always one more improvement to be made. Anyone who thinks otherwise is simply naive.

Andy Lester has issued an interesting challenge regarding CPAN. First, I have to say that CPAN is probably at least one-third of the reason I love Perl. Having a centralized, search-able, browse-able, and documentation-oriented repository of modules makes development so much easier than any other alternative I'm familiar with. In addition to all that, it has reviews, forums, a bug tracker, test result summaries, etc. That said, it is far from perfect.

There are two main complaints that I get. The first is from friends who are systems administrators. They often despise CPAN because they don't care if it helps developers easy locate and include modules. All they see is that to install some application, they have to install 15 modules, which installs another 22 dependencies for those, and then another 18 for those, and then another 12 for those, and then another 5 for those and then finally 2 more. This can take hours to build and install all of them. This is just a hassle for them.

From a developer perspective, I also have the problem that Andy highlights: which effing modules solves my problem? For example, if I want to build Subversion/CVS/Git style commands with sub-commands I could use App::CLI by clkao or App::Cmd by Ricardo Signes. Which one is better for my problem space? Both are by smart developers and both are well put together, but they are completely different implementations for a similar problem. This is something that CPAN won't help you with very much. Andy's example of XML is even more complicated.

There are other problems as well. Here are some ideas I've had that could be implemented in individual chunks that could help.

• Wikifying the documentation. This is an idea I've thought some about and even worked on designing an implementation for a bit over my Christmas break. This is basically about allowing visitors to contribute POD to a module on CPAN using a wiki-ish interface. This could improve the documentation. The challenge is that CPAN documentation is generated out of the POD stored in modules themselves. There needs some way to make sure that the POD updates made on the Wiki site can get back into the modules easily. I developed what I think would be a working solution to that problem, but requires the author to pay attention to the wiki and make sure to download the patches. Build tools could help automate this process, though.
• Dependency mapping to cluster modules. Every module contributes a metadata file that should list the dependencies required to run that module. I can think of several useful things we can do here on a large scale. We can use that to suggest which modules are more popular or better based upon how many modules depend upon it. We can use this to indicate (or estimate) how many overall dependencies a particular module has, i.e., you will need to install 184 module dependencies to use this module. We can also tie this dependency information into other features like reviews and ratings to generate other helpful statistics and heuristics.
• Incorporating use statistics. Sites like ohloh, or iusethis have a good idea in how they can rate content. It's not a perfect system, but letting a person just click a counter that says, "this is good" or "I like this" provides a very simple mechanism to rate a module. In addition, it allows you to gather statistics grouping modules again in another way.
  

None of this is a new idea. None of these are really very innovative, but I think they would be steps in the right direction toward making CPAN work better and better. If we can incorporate additional metrics into how the search works, adding tagging, or who knows what else along the way would go a long way.

I wonder if we could expose the CPAN services currently available in some sort of unified web service that would allow developers to try and enhance and experiment too. This could make it so that CPAN grows in new ways without a lot of overhead to get TPFs attention or what-not.

I don't know. More ideas...

Cheers.

Okay, so now that I have my fancy new blog, it's time to start blogging again. For the past month or so I've been a little short on spare time, but what time I've had to spare has been spent playing with two new toys: Twitter and OpenResty.

Lance got me hooked on Twitter and it's pretty cool. For those that don't know, it's kind of like a Facebook status if you know what that is, but it's more than that. It's also like have a public instant messenger that you send to everyone and everyone you're following sends to you. It's also like having really short blog posts. Anyway, it's kind of cool for those quick things you find and think, "Dang, that's neat, I'd like to share that with my friends." Things I used to post to the K-Slug channel on IRC no go to Twitter. Now, I just need to get more of my friends on there to see them. :)

The other thingy I've been messing around with is OpenResty, which is somewhat of an intriguing solution for serving a RESTful web service. Basically, it's just a web service API that allows you to create database tables, rows in those tables, users, control access to the tables, etc. It provides no UI at all, just the ability to GET lists of or individual resources, POST new resources, PUT updates resources, and DELETE resources. You can then build a front-end to the data using whatever application server you like, client-side JavaScript, Adobe AIR, etc. (If you want the buzzword, this is very "Web 3G"---I'm now looking for a waste basket to puke in.)

Anyway, I'm kind of in search of a problem to solve using OpenResty to find out what it would be really helpful for and I'm just really intrigued by what it offers. But that's what I've been dinking around with lately.

Cheers.

I'm currently experimenting with some stuff related to using JavaScript on the server and the client. The results I'm getting are somewhat mixed. The idea I had was to see about letting someone customize some aspects of an application I'm developing that is intended to help you live more healthy. Basically, based upon the statistics of a food or other parts of the application, you could create custom metrics to measure your success or failure according to your particular preferences. (Some people monitor carbs, some monitor Weight Watcher points, others are interested in fat and protein, etc.)

I've successfully added tools that will allow you to add your custom metrics to the system using your own JavaScript snippet. This snippet is passed a object representing the food item (and eventually to include other information). I can then calculate your metric based upon the current food your viewing and your code.

This calculation is performed on the server side using Mozilla Spidermonkey
and the JavaScript
module on CPAN. On the server, this ends up looking something like this:

use JavaScript;
my $runtime = JavaScript::Runtime->new;
my $context = $runtime->create_context;
$context->eval_file('benchmark.js'); # base API library

# do some other setup of the objects I need in __tmp.food

$context->eval(

     q/function evalIndicator(food) {/

    .$user_calculation # this variable contains code from the end-user

    .q/}/

);

$result = $context->eval(q/evalIndicator(__tmp.food)/);

I've added a little more in there for error handling and such, but that's the core of how I handle the work on the server. The really nice thing here is that the user's JavaScript can't really do much bad other than a denial of service by looping or recursing internally. The server-side JavaScript doesn't include any BOM objects that could be used to do terrible things like steal user information, etc. Furthermore, if I run this code as a separate process that communicates back to the server, I can easily wipe it out if it takes longer than a few seconds to work.

Then, using a combination of Dean Edward's sandbox
and some custom hacking of my own, I can then provide on-the-fly updates from the client using most of the same JavaScript code. This allows the JavaScript to run in a relatively safe container within a hidden IFRAME.

On the client, I've implemented the sandbox in this form. I've left out some of the error handling and other details for simplicity:

// Dean Edward's sandbox in OO form:
function Sandbox(preloads) {
  this.iframe = document.createElement("iframe");
  this.iframe.style.display = "none";
  document.body.appendChild(this.iframe);

  this.frame = frames{eval:function(s){return eval(s)}};"+

    "top=opener=parent=null"+ // cut off access back to the parent

    "<\/script>"

  );

  this.frame.document.close(); // stop the throbbing

}


Sandbox.prototype.eval = function(code) {

  var sandboxObject = Sandbox.functions[this.serialNumber

;

  return sandboxObject.eval(code);

}

Sandbox.prototype.close = function() {

  document.body.removeChild(this.iframe);

  Sandbox.functions[this.serialNumber] = null;

  this.frame = null;

  this.iframe = null;

  this.serialNumber = null;

}

Sandbox.serialNumber = 0;

Sandbox.functions    = new Array();

Now that I have the sandbox to run things in, I need to setup a similar environment as I use on the server. My initial implementation embeds the user's code as a special attribute in the middle of the page, but it could be special block in the header or an Ajax request once to fetch the code string into a variable just after load, etc.

// This code runs onload, to make sure benchmark.js is loaded completely
var sandbox = new Sandbox(['benchmark.js']); // base API library

// When the user changes a value inthe page, the calculation can be made again on the fly...

var result = sandbox.eval(

    ""

  // do some other setup of the objects I need in __tmp.food

  + "function evalIndicator(food) {"

  + user_calculation // this variable contains code from the end-user

  + "}"

  + "evalIndicator(__tmp.food)"

);

Again, I've omitted some of the other details to keep this article at a manageable length. That's the gist of how I run this on the client.

Now, after I've done the work, I probably am going to drop the client-side effort and provide updates by contacting the server using Ajax-ish requests. Why? Well, I can control the server-side environment much more closely by running the code in a separate process that I can kill off if it runs away. If someone's code accidentally (or intentional) tries to do something malicious on the client, I get the blame. The browser is a much less controllable environment. They could fire off requests passing information regarding the end user off to an unknown third party or compromise something even worse if I'm not very careful. I'd have to do a very thorough job of stripping the sandbox area completely. Even then, I don't know if I can actually protect the user from a malicious attacker since there are so many unknowns regarding browser implementations. I'd need to do a lot more studying before I'd be willing to run untrusted code this way for sure.

On the other hand, this trick might work swimmingly in an environment where I trusted all the code being executed in both boxes. I'd certainly be willing to do this on a project at some point to provide calculations on both server and client using the same embedded code, which is why I'm writing the article for others. Anyway, this has been an interesting adventure in JavaScript.

Cheers.

While working at my current job, we've used three different systems to manage our projects. First, we used Trac
, which I liked very much. As we transitioned from project management into user support after the first launch of the new web site and related products we tried to use RT
for project management, which I like very much as well, but turned out to be a poor choice for what we were doing. Finally, over the past few months we've implemented a new system during Drupal
, which has been moderately successful for us. I wanted to discuss the progression briefly and explain how to use Drupal for the task of software project management.

Initially we used Trac. Trac is basically a wiki + issue tracker + code browser + milestone manager. It does those four things extremely well together. You can create wiki pages for documentation very easily. You can easily link between wiki, issues, code revisions, files, etc. You can gather your issues together in dated milestones for releases. You can track all the activity of your developers from a central timeline. All of these were working well for us. The only major drawback is that my knowledge of Python (which Trac is written in) is limited and there are things we wanted to do with it that would have required more customization than Trac is really capable of without patching it. That wasn't a show stopper. I can learn Python and we could have lived without the customizations, but it was a drawback for us.

However, Trac is not a tool that works well for end-user support since it's email integration doesn't work very well and isn't quite as transparent as I'd like. Having used RT for end-user support and loving it for that purpose, we implemented that and transitioned to RT for project tracking as well. At that point, I setup a Drupal site to sit beside the RT site to hold documentation. As far as end-user support goes, RT is the bomb. The keys, as far as I'm concerned, is the ability to logically separate support issues into queues, being able to transparently integrate email, being able to link and merge various tasks together, being able to make private technical comments on a ticket separate from replies that go out to the person with the problem (that's a killer feature), and having lots of flexibility when it comes to building email templates and triggers that send email on comment... oh, and really, really, really flexible access control. I've not seen a system for incident tracking with end-users that even comes close (not that I've seen many such systems). I've also made quite a few customizations to the system to make it more useful for us.

As far as project management goes, the RT-Drupal combo was a failure. It caused a lot of headaches, there was confusion about what was happening, and we really didn't store much of any documentation in Drupal. We should have added RT as an end-user support mechanism and kept Trac in place without modification. Had we done that, I don't think we would have implemented the Drupal system we're using now, however. Basically, I/we decided that trying to use RT to track our software projects wasn't working for us.

I suppose we could have found a way, but we decided instead to give the Drupal Project and Project Issue module a try. This decision was made for several reasons. First, we'd made a strategic decision to use Drupal as the new base platform for our central web service, the Boomer Knowledge Network
. By reusing Drupal for project management, we could more effectively reuse our knowledge and even try a few things out on the support site that could be copied on the BKN. We weren't sure it would succeed and we might have switched back to Trac again if it hadn't.

However, our use of Drupal with the Project and Project Issue modules has worked out better for our purposes at this point than I think Trac would have. The Project module for Drupal provides little to no value on it's own for us except to break up our various software projects into chunks. The Project Issue tracker, on the other hand, has all the power we need. The only major drawback at this point is the inability to assign another user to an issue //drupal.org/node/4354">http://drupal.org/node/4354, perhaps, eventually they will even agree on a fix and apply a patch!
. Other than that, it can track pretty much anything we need it to in a way that fits our team's development style [# Our development style bears some vague resemblance to Agile, but much less formally structured].

The thing that's really helped though is Drupal's CCK module. This is a customization tool that allows you to add custom fields on nodes. The Project module has a system for building releases and such, but it's really strongly geared towards how Drupal's core, theme, and module code is released, which isn't precisely the way we release our work. Rather than using it, we built our own system by building a custom Release node and then associating custom fields to link that node to a project, setting a status (in progress or deployed), a planned delivery date [# usually within 2 weeks], and a list of references to issues that should be completed on release.

We then reconfigured the states that they are defined for issues so that we have a "fixed / ready to deploy" state and a "deployed / completed" state to match with our releases. Those working on a given release can change the state of all the issues to "ready to deploy" as soon as they get everything on that issue complete. Once all of them go light green, I "schedule downtime" [# I.e., I do it late at night when our members are asleep] and perform a code push and do all the configuration necessary to finish the production release [# Usually, this is a 5 minute process, but sometimes as long as 2 or 3 hours]. Then I can mark that issue as resolved.

We also have some documentation in the system, but we're not doing as well here as we probably could. I'm also starting to build some tools using more custom node types with custom CCK fields to handle idea gathering. I'd like team members to be able to visit a site and share that link in a way similar to Digg or Delicious within the office and I'm already a good ways along on that. We're building a review system so that other staff outside the team will be able to comment on screenshots and comps so we can develop a feedback system.

I've also figured out a way to get the timeline back, which was a feature I really liked in Trac. Using the Drupal news aggregator, I can pull the feeds for code commits, issue creation, releases, and even comments [# via the Drupal Comment RSS module]. Unfortunately, I haven't found a way to pull issue followups in a way similar to Comment RSS so those are missing still, but I may just build a Followups RSS module if I have to.

Anyway, I'm pleased with how the project modules are working in Drupal for us. It's a little more flexible than Trac is (IMO) and since I'm very familiar with PHP [# I'm unfortunately familiar with PHP, which is a language I just don't like for both idealistic and emotional reasons.] and with Drupal [# On the other hand, even though Drupal is written in PHP, much of its design both internally and aesthetically is quite nice.], the code customizations are a piece of cake.

Well, that was a lovely ramble. That's what I get for writing so late after working all day....

Cheers.

I'm a big fan of Open Source software. I began to be a fan of Open Source (or Free Software as it was more generally known back in those days) because I feel that it is not right to "steal" software. When I started towards Free Software, I did so because I was a rather poor college student and I couldn't afford much of the commercial software available for various tasks I wanted to do. When I found out that much of that software was available for free as long as I learned to use Linux (and I really enjoy trying new things) I wouldn't have to worry about "stealing" anything to do what I wanted to do.

I switched to Linux, started using free tools for almost everything, and now have come to love Free Software for many of it's benefits. In my own software development, I often release my work under a Free Software license not because I necessarily think anyone else might find it useful but because the quality of my work increases when I do it. Open Source provides accountability.

This accountability may not work for everyone, but it certainly works for me. I am not a person who wants other people looking at my work and saying to themselves, "That's a piece of @#$%! It has no documentation. It has a horrible interface. It engages in bad design practices." The reason for this is two-fold: (1) I am a lot more concerned about appearances than I like to pretend and (2) I use my Open Source code as a resume enhancement and if I put out crap, I can expect my resume to look like crap.

Therefore, I often seek ways to take a program I've written for myself or for Boomer Consulting and see how I can abstract that code out so that it could be used in a my general solution. When I do so, I then go through all the modules and add documentation, I am more careful about adding inline comments to explain what it's doing, and I generally work harder at making my code reusable rather than hacking together a quick solution that just gets the job done. And, perhaps most importantly, it forces me to write extensive test cases to make sure that the code actually works rather than just performing the trial-and-error process one time around and then never verifying that a feature works again until it breaks in production.

In the short term, this often has the tendency to make my life harder (even significantly so), but in the long term, my it means I have to spend less time trying to fix my own crappy code because I was lazy about how I put it together initially.

Anyway, that's just one more benefit of publishing my code as Free Software.

Cheers.

I've been going through the list of tables on my Drupal site in prep to convert an existing single-site to a multi-site with a shared database. However, I haven't found any reference that illuminates the general purpose of the tables. I've tried to put together at least a preliminary list here. I share it in hopes that someone else might find it useful, but it is not normative or complete.

access

User. This is the table used by the "Access Rules" section for banning usernames, IPs, etc.

accesslog

Statistics. This stores the accesslog if Drupal is collecting statistics.

aggregator_category

Aggregator. Store the categories feeds may belong to.

aggregator_category_feed

Aggregator. Links feeds to categories.

aggregator_category_item

Aggregator. Links feed items to categories.

aggregator_feed

Aggregator. Store the feeds that the Aggregator module pulls from.

aggregator_item

Aggregator. Store the items that the Aggregator module has pulled for the various configured feeds.

authmap

User. This stores a map to the authentication module used to authentication each user in the database.

blocks

Block. This stores information about blocks provided by every module installed, including custom blocks.

blocks_role

Block. Stores the roles permitted to view blocks in the system.

boxes

Block. Administrator created blocks.

cache

System. The general cache of data, used by many modules.

cache_filter

System. I have no idea what this table is used for.

cache_menu

System. I have no idea what this table is used for.

cache_page

System. I have no idea what this table is used for.

cache_views

Views. Used to cache information related to views.

client

Drupal. Used to record Drupal client sites. I believe this is the list of sites that others have logged in from if you use the Drupal module.

client_system

Drupal. I'm not exactly certain what this table is for.

comments

Comment. Stores all comments made on your Drupal site.

contact

Contact. Stores the emails sent via a contact form.

files

Upload. Stores information about files uploaded via the file upload module (and used by some other modules too).

file_revisions

Upload. Associated file revisions with node revisions allowing different node revisions to have different versions of a file associated with them.

filters

Filter. Stores information about the body content filters used in your install.

filter_formats

Filter. Associates filter formats and settings with a body content filter.

flood

Watchdog. Used to detect floods from requests coming from a set.

forum

Forum. Used to link forum posts with the forum topic.

history

Node. Used to track which nodes are read/unread.

locales_meta

Locale. Something to do with language translation.

locales_source

Locale. Something to do with language translation.

locales_target

Locale. Something to do with language translation.

menu

Menu. Storage of menu module customizations.

node

Node. The main table for storing general node information, including the title, node number, dates, workflow state, but it does not store most of the actual content.

node_access

Node. Storage of per-node access permissions.

node_comment_statistics

Comment. Notes certain statistics related to the number of comments and how recently comments have been made on a node.

node_counter

Statistics. Records the view count for each node.

node_revisions

Node. Stores information about node revisions, including the main content of the node.

node_type

Node. Stores information about the custom node types.

permission

User. Stores the permissions that have been assigned to each role.

poll

Poll. Extra information associated with poll nodes.

poll_choices

Poll. Associates the available choices with a poll node.

poll_votes

Poll. Votes by visitors on a poll node.

profile_fields

Profile. Definitions of available profile fields.

profile_values

Profile. Profile field values associated with a particular user.

role

User. Assigns role IDs and names to all the roles in the system.

search_dataset

Search. Something to do with search.

search_index

Search. The search index.

search_total

Search. The number of times a given word appears on the site.

sequences

System. The current counter for each of the sequence IDs.

sessions

User. User session tracking data stored in the database.

system

System. Information about installed modules.

term_access

Taxonomy Access Control. Access control per category.

term_access_defaults

Taxonomy Access Control. Access control per vocabulary.

term_data

Taxonomy. Definition of taxonomy terms.

term_hierarchy

Taxonomy. List of parent terms for each taxonomy term.

term_node

Taxonomy. Table linking taxonomy terms to nodes.

term_relation

Taxonomy. Relationships between taxonomy terms.

term_synonym

Taxonomy. Alternative names for a taxonomy term.

tinymce_role

TinyMCE WYSIWYG Editor. Roles assigned to use TinyMCE profiles.

tinymce_settings

TinyMCE WYSIWYG Editor. Definition of TinyMCE profiles.

url_alias

Path. Path aliases recorded to nodes.

users

User. User records.

users_roles

User. Link table between users and roles.

variable

System. Administrative settings and other site-wide variables.

view_argument

Views. Definition of arguments to a view.

view_exposed_filter

Views. Definition of exposed filters in a view.

view_filter

Views. Definition of filters in a view.

view_sort

Views. Definition of sorting in a view.

view_tablefield

Views. Something to do with the views plugin.

view_view

Views. Basic information about a view.

vocabulary

Taxonomy. Definition of taxonomy vocabularies.

vocabulary_node_types

Taxonomy. Associates vocabularies with node types.

watchdog

Watchdog. Records watchdog log entries.

Drupal
is by far my favorite CMS, despite the PHP (or poohp as I tend to pronounce it). One of the places Drupal is strong is organization, as long as you want to organize everything in a web. It does provide hierarchies and has breadcrumbs, but these are where Drupal begins to show weakness. In this articles, I'm going to explore how Drupal organizes itself and how you can take advantage of that organization out of the box, or with some module help, or by custom hacking. It's a summary of the research I've been doing on the topic.

Organization

Organization of content in Drupal, as shipped, primarily operates in these ways:

• Follow that term. You click on a post in a list and see a list of categories (formally "terms" in Drupalese) associated with the post. You click on one of those and you can see all the other posts on the site that are like that one.
• Order from the menu. You can assign a page to one or more menus, which are just hierarchical lists of links. If a menu item on the screen matches what you're looking for, clicking on it will take you to the node, term page, a list, or whatever the link points to.
• By the book. Pages can be arranged in ordered hierarchies, like books. If you look at a page in one of these organizations, you can click on the sub-pages or go to the previous page or the next page or up to the next level from any page here.
• Follow that forum. Forums are really glorified category terms.

That's it. Those are all the organization options you have built-in.

Caveat Emptor

However, there are some inherent gotchas that newbies aren't always aware of. The first is that the breadcrumb is pretty weak in Drupal without some help. This usually reveals itself in a scenario like this (this is approximately how I ran into it). A newby downloads and installs Drupal, logs in as administrator, and goes to create his first vocabulary (a custom system of categories). He creates a hierarchy and adds a bunch of terms. He finishes and adds a test node that has one of the category terms set that has two or three parents. Let's say, "Food > Fruit > Citrus > Orange". However, rather than seeing a breadcrumb on the node view that reads "Home > Food > Fruit > Citrus > Orange", he just sees "Home". If he's like me, he's confused but just assumes he did something wrong. He tries clicking on the "Orange" term to see the breadcrumb there. The breadcrumb turns out to be exactly as expected, "Home > Food > Fruit > Citrus". That means if someone gets that term and clicks on the test node, the breadcrumb that should, intuitively, expand one level deeper completely evaporates.

The problem is that this is a misunderstanding of the purpose of the taxonomy system. The taxonomy system does not provide any structure for the nodes. It provides some structure for itself, but the nodes themselves still belong at the top. It's a completely flat structure. Structure is provided to nodes in Drupal core in one of two ways: the Book module and the Menu module. Which way is appropriate for your site will depend on your needs. Also, neither of these may be satisfactory, but we'll get to that in a bit.

Book Solution

The first mechanism provided by Drupal is the book module. This module allows any node to have a parent. To use the book module, you must first enable the module, configure the permissions, and then create a Book page. The first Book page will be at the "Top Level" meaning it starts a new "book". You can then create additional nodes that are in the tree. The top must always be a Book page node type, but the rest can be any node type you like, but you have to go through the extra step of configuring the node via the Outline tab after your create it. Update April 21 @ 7:28pm: I just realized this is an error. I forgot about the Outline tab. You can make any page the root of a book via the Outline tab. (Book pages automatically have the "Parent" option configured on their create form.)

If you look a the breadcrumbs of Book pages, they operate exactly as expected. Each breadcrumb is created by traversing the list of parents until it gets to the top level and then adds "Home" in front of everything. The book module can also provide a hierarchical list of pages that looks like a typical Drupal menu as well via the Book block that comes with the Book module.

This is, by far, the most common sense method for hierarchically organizing your site's content. However, if you need special content other than the book pages you have an extra step to make it work. If you need to organize non-node content (special forms or something) the Book module may not be any help at all. Also, a single node can't belong to multiple books. A node may only have 0 or 1 parents in a simple hierarchy.

Menu Solution

An even more flexible solution is through the Menu module. The Menu module allows you to create Menus, which are hierarchical lists of links (it allows you to manipulate the built-in menu items as well, but that's another topic). These links may be to nodes, terms, or anything else on the local site that has a URL. They may also be absolute links to get to locations on other web sites. Each menu item has exactly one parent, but since you're only creating links, you could have a single node or other item linked in as many different menus as you need.

Furthermore, creating a hierarchical menu also tweaks the breadcrumb. If you have a node that would otherwise appear at the top level (i.e., it doesn't belong to a book), the first menu it is listed in will be used to create a breadcrumb for that node. Therefore, if you had a taxonomy structure with a matching menu, you could fix the problem of having your node appear without a breadcrumb.

Unfortunately, this too comes at a certain cost. You must maintain the menu structure manually. You have to add each node as it goes. If you have a taxonomy structure to mimic, you have to do double duty on your data entry and then make sure your menus point to each new node you create.

Setting up the menu while creating a node is pretty easy for an administrator. Anyone with permission to administer the menus can setup the menu for a new node via the "Menu settings" box on the create page. However, if you want authors to be able to do this without being able to tweak the entire menu structure, you're out of luck. You either grant someone the right to manipulate everything or nothing, which isn't acceptable in many situations with more than a few authors.

You can address this issue by looking into the Menu Subtree Permissions
module. This allows you to grant edit access on parts of menus to certain roles.

Add-ons

There are many, many add-on solutions for handling this. I'm going to give my favorite choices and then list the others I've found. This is not a complete list, but it does highlight the modules I've examined a bit in my search for a solution.

Taxonomy Menu

A solution that uses the combined power of taxonomy and menu is available in the Taxonomy Menu
module. This module creates a menu item for each taxonomy term in the system. This automatically gives you both downward navigation through menu items and upward navigation through breadcrumbs. This isn't quite as good as the navigation provided by the book module, but with intelligent use of the sticky flag or manual items, you can come relatively close.

Add on Custom Breadcrumb

Another way of handling the breadcrumb is via the Custom Breadcrumb
module. This module allows you to associate a breadcrumb with a node based upon it's node type. For example, if you have event nodes that always needed to appear in "Home > News > Calendar > Events," then you can use the Custom Breadcrumb module to do this. If you need a more variable structure than this, then Custom Breadcrumb still doesn't do it.

Others

Here are some other add-on modules that you may wish to consider:

• Taxonomy Bradcrumb
  . This module provides half the abilities of the Taxonomy Menu in tweaking the breadcrumb. It does not, however, provide any way to navigate down to the items in the tree, just the parents on the way back up.
• Category
  . For the daring, there's also the Category
  module which provides a complete replacement system for the core Taxonomy module, Book module, and provides a special menu module for creating menu items automatically on the creation of nodes. However, this is a big module with lots of options and I've back off from using it because it's almost too powerful to figure out how to use it most effectively. I've had some trouble getting it to work really well.
• Pathauto
  . This module doesn't actually help you organize your site, but it can makes your URLs match your breadcrumb organization structure. I highly recommend this module to make your URLs pretty. However, be aware that the URL aliases can quickly clutter the URL Alias page if you have a lot of pages.

• Theme Hacks

  If you do a search on the Drupal site for "breadcrumb" you will come across several hacks to the theme. All of which are customizations that make use of drupal_set_breadcrumb()
. You pass that function an array of links and it will give that array of links to the page template. This is the ultimate solution if no other will work for you.

  However, again, this still only gives you the upward navigation. It doesn't provide any kind of downward navigational structure.

  Conclusion

  Drupal does provide ways for organizing your content, but some of the out-of-the-box methods may not be ideally suited for your situation. Drupal comes out of the box with excellent support for web-like and river-of-news style organization. Hierarchies are quite possible and can be managed, but the structures provided for such assume a relatively light amount of hierarchical information. However, by using a few modules and the taxonomy structure, you can create very flexible and functional hierarchies to suit any site structure.

We made a major decision this week that I'm, personally, very happy about. The Boomer
web site is moving completely to Drupal
. Currently, our site uses a CMS called Magnolia
, which is an excellent product, but the needs of our web site are drifting out of sync with the features provided. Magnolia provides good content management, workflow, and document management, but our site is growing in the direction of community building rather than publications. We're already using parts of Drupal for part of the Boomer Extranet service for our top-tier clients
. We've got another offering coming soon that will make Drupal an excellent fit for us and it just doesn't make sense to maintain two platforms anymore.

Since we're still developing business plans and this product is not officially announced, that's about all I can say for now about the community part of the web site. However, I would like to discuss some of the issues I'm going to be dealing with in moving away from Magnolia to Drupal. Most of this involves handling the publication aspects we do deal with in Magnolia that need special consideration when moving to Drupal.

For example, we have a reference library of articles related to topics specific to the accounting profession. We provide these articles to our premium site members, but provide only stubs to the articles to the public. Therefore, I need a way to store these articles, categorize them, and associate downloadable documents with them. I also need a way to secure access so that only the teaser and tags are available for marketing purposes while the rest are avaialble only to the premium subscribers.

Since this is a big issue to cope with and one that Drupal doesn't directly deal with (though I considered a possible solution in an earlier post
), I've created a Drupal Group
to deal with the topic of document management
. I hope to find others interested in the topic to create better solutions in Drupal for this problem. Anyway, if you're interested, please join
the discussion.

Cheers.

There's an old saying that goes, "If it ain't broke, don't fix it." Scott Adams once made the comment regarding how engineers think, "If it ain't broke, it doesn't have enough features." I think the latter thinking must have applied when Google released their latest Google Notebook
improvements this week.

I've become a major fan of this tool over time, but the new look is clunky and cluttered. I liked how sleek the interface was before, but now there's a bunch of extra shadows and rounded corners and borders that really serve no purpose but to distract me from my note taking. The notes also don't collapse very well anymore. I've also found some bugs in the search.

Bah. I hope they fix this soon.

Okay, in my last post I posted a blog from Microsoft Word 2007 straight into my blog using the blog API that the latest version of Word supports. I've known I could do this from Google Docs for a bit now, but I hadn't gotten it to work. The reason I hadn't gotten it to work was the same reason that I had trouble getting Microsoft Word to work, the Blog API for Drupal doesn't really work unless the Blog module is turned on. Normally, I blog on my site using the "Story" module rather than the Blog module. However, once I turned on the Blog module, I was able to create Story nodes from Word and now from Google.

Google Documents killer features include the following:

• Easy to use. Anybody can figure this thing out. Login, create a document, edit the document, save. Done.
  
• Easy to share. You can share your document with anyone who has an email address. They can then login, edit your document, and save.
• Easy to compare. After you have edited your document or others have too, you can see every change that has been made by clicking on the Revisions tab. Again, super-easy.
• Spreadsheets too. Many of these same features are available in the Spreadsheets as well as the documents.
  

It's not so great in that some of the editing tools provided are a little too simple. There are no custom styles. There are only three kinds of headings. Tables are difficult to use. Bulleted and numbered lists can't be formatted (outlines are very difficult). You can change fonts, text color, font size, and some other aspects, but there's no way to use styles to make this consistent.

Finally, the worst part is that some of the HTML generated is not clean¿i.e., it actually includes things like font tags and uses line breaks heavily rather than paragraphs. The new Word publisher made my blog entry completely clean. I didn't really use any formatting so it wasn't a complete test, but it was still much cleaner than I've come to expect from Word.

Anyway, I'm using Google Docs heavily at home, especially for collaboration on my pro bono projects. At work, once Word is in place with SharePoint and the other tools to back it up, I think that will be the collaboration location of choice there. We'll see and I'll keep you posted.

Cheers.

For a three years now, one of my favorite IT tools on my bat-belt has been RT
from Best Practical
. RT (Request Tracker) is just another bug/incident tracker, but it's got a lot of great features like seamless email integration, highly customized access control, a self-service web site, and the ability to make public replies and private comments on any ticket in the system (allowing you to keep the shop talk as technical as you need to record solutions while only giving the end-user an overview of the solution).

I noticed in the last couple months that Best Practical has created a new service web site called Hiveminder
, which is basically just a to do list manager. However, as to do list managers go, it's pretty nice. I've been using it for a week now just to see how I like it. The jury is still out on that one.

Of course, I'm never satisfied with just seeing a service, I wanted to know how it worked. I initially thought that Best Practical might have adapted their RT to handle to do lists (which wouldn't be very hard) and added some nice JavaScript front-end enhancements. That is not, however, what they did. They've been building a web development framework called Jifty
on which Hiveminder is built. This isn't really news, but it was news to me so I added it to my list of things to look into when I had the time.

This weekend I decided to finally install it and give it a shot. So far it's wowed my socks off. Before this web site became just another part of my blog, I was working on a web framework. So far, Jifty is the web framework I would have written if I'd really had the time and experience to make it happen. I'm building a very small image gallery application to see if I can get something more to my liking than anything I can fid that's already written. I'd say that after a few hours (thoroughly interrupted by feedings and calming a 1 week old) I'm well on my way to something very nice.

If you're familiar with DBIx::SearchBuilder
(used extensively by RT and, I believe, written for it) or with the internals of RT, Jifty will look familiar as well. They've reused a lot of similar ideas. For example, Jifty::DBI
is the database development framework, which is pretty much a rewrite of DBIx::SearchBuilder with a lot of the functionality that RT extended it with and an extensible new way of writing database schemas. Jifty itself does almost everything for you automatically and then you can choose which parts of the built in functionality to override. It provides some nice tools for making your site look really nice and have that all-in-one-page-Ajax-feel.

Anyway, I'm pretty excited to see what I can do with it. If I successfully put together the image gallery application, you can look forward to seeing it here sometime during the next month.

Cheers.

I run Windows on my Linux desktop using VMware Server. It works pretty well. However, one thing I hate about VMware is the fact that it has a direct key binding to reboot the system, without confirmation. The key binding is to one of my most oft-used keys: Ctrl-R. I hit Ctrl-R on a very regular basis because that's the hot key to reload a page in Internet Explorer and Firefox, which is something I do pretty frequently as a web developer. Anyway, since I mouse back and forth between VMware and my Firefox browser, about once in every 50 Ctrl-R's, focus is occasionally back on VMware, which causes an immediate reboot and also an immediate expletive from me.

This is stupid and, from reading the VMware forums, a problem that's existed since at least 2004 that still hasn't been addressed. So far the solutions are either to make it so that it's really hard to give VMware focus by accident or to turn on hints and enabled the hint on resets to add an additional confirmation when you make this action. Both are unsatisfactory.

However, in case you are also frustrated by this, I can keep you from going further to look for a solution. Close your VMware Console and edit the preferences file, which is ~/.vmware/preferences under Linux. In that file, add a line reading:

pref.vmui.reset = "TRUE"

Next, save the file and open VMware Console back up and go to Help > Hints > Show Enabled Hints. Make sure you do both or you will still be frustrated the next time you hit Ctrl-R.

Cheers.

As a fan of Perl, I'm very used to lists of items that are written with an extra comma. For example:

my %hash = (
     foo => 1,
     bar => 2,
     baz => 3,
);

Even though there's not a fourth item in the list above, it still works. This is commonly referred to as the "trailing comma." Switching languages, the trailing comma is supported by Mozilla's JavaScript 1.5 implementation
. It is not, however, supported by IE (not even IE 7).

Therefore, when I perform a very similar operation as above, I have a nasty habit of forgetting this and breaking my code in IE:

var hash = {
    foo: 1,
    bar: 2,
    baz: 3,
};

The above runs fine in Firefox, but will fail to run with an error in IE 7. When supporting IE, you (I) must remember to remove the trailing comma:

var hash = {
    foo: 1,
    bar: 2,
    baz: 3
};

Okay, so some newb is probably reading this wondering, why is the trailing comma valid at all? Simple. It makes it just a little easier to add another element to the list later and even to reorder with copy and paste line-by-line. It can save just a small bit of time coding. That is, I could paste "qux: 4," into the first JavaScript example above foo, below foo, below bar, or below baz without worrying about it if I am able to use a trailing comma. In the second example, I can insert it above foo, below foo, below bar, but not below baz. Furthermore, after adding the comma after "baz: 3," I still cannot put it after baz until I remove the comma from the end of the pasted "qux: 4." This might save me only 20 seconds in my coding of paste, click, comma, click, backspace, but a little thing like this goes a long way in helping my sanity.

Anyway, I just thought I'd post this real quick in the hopes that I'll remember next time.

Cheers.

I previously published an article here about using Drupal and RT together to create a knowledge base system. I've continued to improve the system and wanted to give an update on how it's working.

Of the pieces I mentioned in my previous article, I have implemented Organic Groups in the system. I have not (yet) implemented any of the other ideas I had at the time. In addition, I have used the Front-Page module, added use of the Content Construction Kit (CCK), and started adding some tweaks to RT.

Organic Groups

I have added Organic Groups to the site to operate the same kind of functionality as is provided by a SharePoint sub-site or a Trac project. Each Organic Group is a smaller segment of the whole knowledge base dedicated to a specific project or knowledge area. Organic Groups allows each user of the site to be a member of just the groups that are of use to him. It keeps membership lists for each group and also provides (some) notification regarding new content (it would be nice if content updates could be included as well, but does not at this time). Content may also be associated with no group or can be associated with a group while also being public as well.

All in all, this has been very helpful to segmenting the site.

Front-Page

I've used the front page module to setup a simple comment that goes to all anonymous users. The knowledge base is available from the Internet, but is only intended for internal use at this time. However, it is possible that someone might stumble upon it (most reading this could probably guess it's location in less than three tries and probably a majority only in one...). I wanted a special page facing out for anonymous visitors so that it would tell them to bugger off (though, in much nicer terms than that). It would also help our staff find the pages they are looking for more easily.

The front page module allows me to setup a front-page that's special to any anonymous visitor as well as having a page that's special to those who have already logged in. After you login to the site, the new front-page is a basic table of contents and list of policy documents that affect all of the group sites.

Content Construction Kit

The CCK is the future of all content on Drupal sites, as far as I can tell. Soon, the "story" and "page" and "blog" content types will just be custom content types created by the CCK. This is a good move. Custom content types has been a weakness of Drupal for sometime and flexinode wasn't quite there—in fact, it was one of those demo-turned-indispensible modules that every programmer dreads creating or maintaining.

In this case, I've used a combination of the CCK, the Category module, and the Event module to create a special content type, the "sub-project". These are a combination of "component" and "milestone," which seems to fit our web development project. Some of the features on the site can be grouped into convenient sets that all can be accomplished together. Each of these tasks are related and often feed each other, but aren't really the same task, thus are handled in separate tickets. This has been very helpful.

Each sub-project has a title, a description, and a due date (the need for the event module). These are plottable on a calendar via the event module. These are also "categories" via the Category module, i.e., nodes that act like taxonomy terms. This allows knowledge base content to be classified into these special categories. This is very cool. I have also pulled these into RT, which I will cover in the next section.

In addition to the above, I plan on creating some other custom content types for various bits. For example, I would like to build a "computer" content type to describe individual machines on our network and a "software package" content type to describe computers. These can then be used in the IT group to associated particular articles with particular computers and software packages. Thus, the CCK content types begin to take up the role provided by custom lists in SharePoint. These can also figure into the RT modifications I'm about to describe.

Auto-Complete in RT

I need to formalize this a bit better, but I have added customizations to the custom field rendering component in RT so that I can make custom fields autocomplete in the same way as they do in Drupal. I have added a "Sub-project" custom field to the Web Development queue. Then, I tweaked the component so that those custom field editors pull the available values from the available milestones in Drupal. This is the first real communication I've added from one system to the other and, so far, to work it requires you to be fully logged in to both systems (no single sign-on yet).

I'm still pretty happy about this and it certainly beats the pants of Microsoft SharePoint. I hope to have further updates as well. Currently, I'm thinking of more CCK types (as mentioned), pulling tickets into Drupal from RT so that categories list the tickets that belong to them (possibly using RT's RSS features and the Aggregator module or Aggregator2 module or some other variant), and getting single sign-on working so that we don't have to login twice anymore.

I'm also still considering some of the things I mentioned previously, though they aren't quite as high on my radar right now.

Cheers.

I've participated in a few Open Source software projects. Some were run by individuals, some by communities, and some by corporations. I'm interested just now in discussing the ones run by corporations. Corporate Open Source holds a somewhat suspicious niche in the market. I'd like to consider here my thoughts on how a corporation can Open Source products and cultivate a community to help develop them.

As a whole, the business community has come to see some value in Open Source. This is particularly true in specific markets where a product or collection of products have really proven themselves, Apache is usually the first to come to mind. I think businesses are quite willing to use Open Source where it is proven, willing to consider Open Source where it looks promising, and willing to develop Open Source where there is potential. However, there is always an undercurrent of skepticism associated with Open Source because the monetary model isn't quite as clear cut as the usual "I'll give you this, if you give me $XXX."

On the other side of things, communities spring up around any successful Open Source project. The quality of those communities is determined by the skills of individuals that makeup of those communities, the attitudes held toward the project, and how well the community works to encourage participation by newbies and gurus (and all the betweens) alike.

Corporate Open Source has a unique position because it melds a more traditional buesiness development model (hierarchical, strict requirements, design, and implementation, milestones, etc.) with the more chaotic Open Source model (chaotic, contributions based upon individual interest or need, few/no requirements, vague deadlines, etc.). If the project is successful, the corporation will also be bolting on a community following which will help direct, design, implement, debug, and test the product. I believe this is a Catch 22 situation: the corporation cannot have a truly successful project without developing the community and the community cannot grow without having the corporation properly directing a successful project.

I see the keys to community development as being wrapped around two key ideas: ownership and communication. I'll start by describing communication and finish with ownership.

Communication in an Open Source project has to be open and honest. This openness and honesty must be percieved more than it must be real. I'm not suggesting that it's a good idea to hide things from your community if you can still make the community think you're honest. I'm telling you that you don't have to dump out the contents of your closets for them if they believe that you are being open and honest---i.e., earnestly telling them enough.

Therefore, keep the marketing people out. At least keep the typical marketing techniques for making your product and company look shiny and perfect out. Just as corporations tend to be a bit mistrusting of the Open Source business model, so are the developers mistrusting the company that has given them something for nothing. The corporation must work extra hard to build trust because both sides will start from a point of skepticism and that skepticism will always remain close under the surface.

Which leads me to my next idea: ownership. You can dispell much of the skepticism of the community by giving them ownership. This doesn't have to be literal ownership of the copyright, but give up enough control in the direction of the product that the community has an important say. If you're planning on a new feature A, but the community is clamoring against feature A to get feature B, consider giving them B first (or instead). If the community contributes a feature that your company can't get any direct value out of, but the community at large loves, consider including it in the main distribution and adding support for it anyway. These are ways of giving the community a sense of ownership. It should be noted, however, that if the community finds a feature valuable, you and your paying clients probably will too.

But why all the trouble? Why would I bother to invest in developing a product just to give it away? The value is that if you develop a "fan base" in the community, you can get improvements, ideas, and testing for free. The opportunity exists for a member of your community to suggest an improvement or provide a patch that links your system up with another system that happens to be a match made in heaven, but that your own staff might not have found while doing research in their spare time---the Internet is a pretty big place after all.

In the case of Boomer Consulting, we sell service. Clients pay us to help them figure out what they can improve, connect them with resources they might not find on their own, and to connect them with other companies who've succeeded in the projects they're still looking forward to completing. I've begun making some contributions to some Open Source projects and we very much plan to make the effort required to release those to the public. These components aren't going to be useful on their own (software never is until someone works to install it, configure it, and use it) and they won't include any of the business logic specific to Boomer that makes us valuable to our clients. We'll be giving away infrastructure that will require considerable investment to really use to compete with---not because the tools are incomplete, but because software doesn't magically start working for you just by installing it.

Yet, if we can get others interested in how those tools work, we can get infrastructure improvements for free when someone writes a patch to do X better or Y to link in another piece of software. All in all, I believe Open Source to be a win for where I work. I'm looking foward to what we can produce during the next year or two.

Cheers.

Phew! This is getting nuts. I am more busy on web development at home than I am at work. Frankly, with this flurry of stuff going on at home, I'm starting to get a bit bored with work. Well, I do want to talk about something I did at work yesterday because it was pretty cool. I've put together a knowledge base and issue tracking system using Drupal and RT. It was pretty easy and I think others might benefit from at least some aspect of the solution.

We've been using Trac to keep track of our knowledge base, issues, milestones, and source browser surrounding the web development project. For the internal IT stuff, we've been using Microsoft SharePoint. Both have had limited success. Trac worked perfect until we needed a way for web site visitors to send in support requests via email. Trac's support for this is total crud. It's still being worked upon, but after having used RT in CIS, I'm pretty hard to please when it comes to support tracker email integration. Sharepoint has worked about as well as it ever does: good enough, but it's a bit tiresome to use. If we were using the beta of the new version, it would probably be fine, but that's not trivial upgrade to perform without causing massive alterations to lots of other services we run.

Okay, so I've started to put together something that could potentially replace both of these with flying colors. It's still a work in progress, but I think I've already got something better after only a couple days of working on the problem.

First, I installed RT 3.4. After installing it and getting Sendmail configured on our hosted server, I got RT configured. I leave learning how to do this as punishment for the reader.

Then, I went in and installed Drupal 4.7 and started configuring it. First thing to do is to install all the modules. I've installed the following:

• Category. This is one of my new favorites. It is a drop in replacement for the Taxonomy module that makes every vocabulary and every term into a node. You can even use arbitrary content types as vocabularies and terms and can have vocabularies within vocabularies. It's very nice. Oh, and you can set the default item depth of a category so that it automatically aggregates children to a certain depth. Also very nice.
• Interwiki. This is kind of the best available for the functionality I'm looking for, but it needs a bit of work to get it right. Basically, you can register prefixes with it that will then be translated into URLs using a wiki-like syntax. For example, it comes configured with the prefix "w" automatically linking to Wikipedia articles. Thus, you can insert Nichols Hall and it would link off to Nichols Hall at Wikipedia. You can add other prefixes to other sites or use no prefix, i.e., foo or //drupal.org/project/pathauto">Pathauto. This ought to be a standard Drupal module. It generates paths for a node according to settings you set. I use it heavily on this site now and it's what allows us to have nice wiki-like articles on the new knowledge base.
• TinyMCE. This is a must have if you have non-technical or semi-technical staff helping you write articles. We have a content editor, Doug who understands but doesn't really converse in HTML on a regular basis. There's no need to make him when TinyMCE exists and is so easy to install and use.

That's it. I installed all of those and then went through the tedious process of setting up access controls, updating settings, etc. Now, I can create a page on the site titled "You are a foo foo" and then create another article and link to it by adding the text [:You are a foo foo
. I can use TinyMCE to format the text without worrying about learning the Wiki-syntax-flavor-of-the-month and so I won't be too afraid to show someone completely non-technical how to enter knowledge base information later on.

I added new custom interwiki prefixes for getting at RT and Trac tickets and the old Trac knowledge base articles as well for the transition period. Now, I can link to our old standards document via BoomerStandards
or to a ticket in the old system via 435
or to a ticket in the new system via 219
. I added a number of other shortcuts as well and this is really the only wiki-syntax anyone needs to know about and this is a pretty easy one to remember.

The knowledge base is now in place. Let's go back to RT. To RT, I've now added a callback to the system for the ShowMessageStanza ticket element in the RT system. That is, I created a file named "Callbacks/boomer/Ticket/Elements/ShowMessageStanza/Default" in the local hacks directory for RT. You may wish to see CustomizingWithCallbacks for more information on creating customizations using the RT callback system. Anyway, the contents of this callback is as follows:

<%perl>
$$content =~ s{(?<!&)\#(\d+)}
{<a href="${RT::WebURL}Ticket/Display.html?id=$1">#$1</a>}gx;

my %config = (

        tracwiki => 'http://trac.example.com/trac/foo/wiki/$1',

        trac => 'http://trac.example.com/trac/foo/ticket/$1',

        rt => $RT::WebURL.'Ticket/Display.html?id=$1',

        http => 'http:$1',

        https => 'https:$1',

        '' => 'http://supportkb.example.com/$1',

);

$$content =~

        s{

                (!|\\)?         # stop now if there's a ! in front

                (

                \\

\|]*):    # match a prefix like abc:, remember the prefix

                (([^\

]+))? # match a pretty name like "Blah Blah is cool."

                \]              # match closing ]

                )

        }{

                my ($escape, $match, $prefix, $link, $title)

                        = ($1, $2, $3, $4, $5);

                if (defined $escape) {

                        qq($match);

                }

                elsif (defined $config{$prefix}) {

                        my $url = $config{$prefix};

                        my $term1 = $link;

                        $term1 =~ s/\ /_/gx;

                        my $term2 = $link;

                        $term2 =~ s/\ /\+/gx;

                        my $term3 = $link;

                        $term3 =~ s/\ /%20/gx;

                        my $term4 = $link;

                        $term4 =~ s/\ /-/gx;

                        $url =~ s/\$1/$term1/gx;

                        $url =~ s/\$2/$term2/gx;

                        $url =~ s/\$3/$term3/gx;

                        $url =~ s/\$4/$term4/gx;

                        $title = defined $title     ? $title

                               : $link =~ /^                      "$prefix:$link";


                        qq(<a href="$url">$title</a>);

                }

                else {

                        qq($match);

                }

        }gex;



</%perl>

<%args>

$content

</%args>

If you're unfamiliar with Perl and Mason, you may have a migraine by now. You might have one anyway, unless you're a JAPH like me.

Anyway, this replicates part of the functionality of interwiki within RT messages by converting the [rt:123
sequences into links. This also features the ability to create links between tickets using a #123 syntax as well, which I'd like to add to the knowledge base eventually as well because I find the Trac-style linking of #123 (tickets), @123 and r123 (revisions), and [123] (changesets) to be very easy to remember and very handy.

That gets me much of the way to an integrated Drupal/RT knoweldge base/issue tracker system. I still need more though, but I haven't gotten there yet. Stuff I'd like to add in:

• Organic Groups. This would allow us to have a subsite for each knowledge base: one for internal IT, one for web development. Later, we could add one for sales information, one for content style guides, or whatever.
• Fix/replace interwiki. The interwiki module is nice, but isn't quite nice enough. It would be nice to have additional formats available. With very little effort, I should be able to create a custom filter to either extend this functionality or outright replace it just for our needs.
• Single Sign-on. I have some experience with this lately. If I can get it so that I only sign in once to Drupal and or RT and then I'm in both, that would be great. If I could implement it using an independent LDAP server, all the better.
• Configuring RT. Configuring the interwiki-ish filter for RT is done in code. This isn't ideal. With whatever solution I work on for Drupal, it would be nice if the RT filter could load it's configuration from the same place, or at least get it from a config file and not in the code itself.

Will I get all this done? Probably not, but I'll get some of it done and if I think it's interesting, I'll try to share it with you.

Cheers.