I was asked the other day why I prefer real paper books over electronic books the other day and realized I’d never really posted on this subject, which I feel somewhat strongly about. I have a fundamental distrust of computers and technology. That may seem a little odd coming from a software developer, but the more I learn about software development, the more this distrust grows. When it comes to books and printed materials, I can condense this distrust into three basic problems:

  1. Digital technology changes too much.
  2. Digitized data is fragile and transient.
  3. Digital printing is not self-sufficient.

Ever-Changing Technology

Ever tried to open a document you created 10 or 20 or 30 years ago? Did you succeed? Chances are you didn’t. Ten years ago, data was saved on 3.5” floppy disks or a CD-R if you had the money for a burner. (The USB drive didn’t come into being until 2000.) Do you even have a 3.5” floppy disk drive in your house? Is the disk or CD-R even readable anymore (since the usual half-life of such things has already past)?

Twenty years ago, you would have used a 3.5” or 5.25” disk. Have you even seen a 5.25” drive in the past 10 years?

Thirty years ago, you would have used… who knows what. I was two years old. I’d have to look at Wikipedia to even get a good guess going. Audio cassettes on your Atari? Yeah, like you have something to read one of those.

Even if you read the media, do you have a program that works anymore to read the format? Do you have something that will read whatever word processor you had then? This is possible and maybe even likely, but even so there’s the risk it won’t work or you don’t have one. Formats change. Programs must be made to read a format. If you don’t have any such program, you might be able to get it to someone who can extra something out of it, but it’s probably gone.

However, I can pick up a book from any of those eras and read them today. A notebook, a journal, a magazine, a newspaper are all quite readable with modern day technology. If you printed your document and placed it somewhere dry and clean, you can still read it.

Fragile and Transient

I already mentioned that the old media fails rather frequently, but it gets worse. Digital data is, by necessity, encoded in an unnatural way. That is, digital data is very robust and easy to process is many ways, but the way we choose to represent digital data is made up.

To demonstrate this problem, let’s consider a manuscript that someone finds in the desert from 2,000 years ago. Let’s say that it’s in a script that no one has ever seen before, will it be understood? Actually, it might be. If that script happens to be similar enough to a known script, a linguist might be able to extrapolate at least something about the contents of the document. All languages and scripts are related, even constructed languages like Esperanto.

Now, imagine the same scenario 2,000 years in the future. Someone picks up a magnetic tape that still has some good sections on it. In the future, they know something about ASCII encoding, but not some of the other encodings that were used today. This one is one of those other encodings. They’d have to use some sort of statistically analysis and hope it’s written in a language they know with text strings they can find or something. They’re going to have a much harder time of it reconstructing the data.

Digital formats add another layer of encoding upon already encoded information. Usually multiple layers of encoding are involved. This makes maintenance and upkeep much higher than for books.

A printed book’s encodings are just the script and language it’s in without the extra layers of encoding, file formats, compression, etc. Digital data has many more things to fail along the way to prevent easy reading in the future.

Self-Insufficient

This is really the most important reason I like books. I can pick one off the shelf, open it to a page, and immediately examine it’s contents. If I have a book in my computer, I have to boot the computer, load the right program for reading it, and read it. If I put it on a flash drive or SD card, I have the full contents of the text with me, but I can’t do anything with it until I get a computer or cell phone or a Kindle or something capable of showing me the data.

A book requires no translator or reader to explain it to me. It requires no special equipment to read other than my hands and eyes.

In the end, a book is something I am sure I can pass on to my children as their inheritance or share with someone else without any additional effort. I rarely have that kind of guarantee when sharing a digital file or something. The digital copy of a book gives me the ability to search or process data in it, but if I just want to read, reference, or consume the content of a book, the printed page is always better.

Cheers.

Recently, I’ve started backing away from Jifty as my web framework of favor. There are a few reasons for this, but mostly it’s related to performance. Jifty is includes everything, Ajax, dispatch, ORM, Comet/PubSub, REST, forms, HTML, templating, and a pony (probably a kitchen sink too). Basically, with Jifty you don’t need to spend very much time working on building up your own utilities or framework upon framework to fit your style or needs. Jifty hands you a style as well, one that I mostly like.

However, the cost to performance on my Qublog project was too much. It was easier for me to port it to Catalyst sans-Ajax than it was to try and correct the performance issues. In addition, I no longer use Jifty for work, wanted to use features of DBIx::Class, Moose, etc. I’m not trying to bag on Jifty here, just explain why I’m moving away from it general. I very well may find a use for it for something in the future.

There is, however, a major feature of Jifty I miss after having moved to Catalyst. Jifty has a collection of classes for encapsulating form rendering and processing. At the center of this is the action class, which is really a glorified functor, but with all the accoutrement required to render an HTML form and to clean up, verify, and process submission. I like this. I like this a lot. I decided to use the flexibility afforded by Moose with an implementation of something like these action classes to create a new form handling system. Over my Thanksgiving vacation, I was able to build up what has become Form::Factory.

An Example

Okay, I thought about showing you the process I worked through to get here, but you probably want to see it in action first. I’ll work backwards from there.

In order to use Form::Factory, you must first define an action class. Here’s a simple login action you might use to present a login form to your users in a web application:

package MyApp::Action::Login;
use Form::Factory::Processor;

has_control login_name => (
    control => 'text',
    features => {
        required => 1,
        trim => 1,
        length => {
            minimum => 3,
            maximum => 20,
        },
        match_regex => {
            regex => qr{.@.}, # really naive check for email address-ish thing
            message => qw{your %s should be an email address},
        },
    },
);

has_control password => (
    control => 'password',
    features => {
        required => 1,
        trim => 1,
    },
);

has c => (
    is => 'ro',
    isa => 'MyApp',
    required => 1,
);

sub run {
    my $self = shift;

    if ($self->c->authenticate({ name => $self->login_name, password => $self->password })) {
        $self->result->success("welcome back,  @{[$self->login_name]}!");
    }
    else {
        $self->result->failure('the login name and password you entered is wrong');
    }
}

The has_control calls establish the inputs expected for our login action and tell the action what features those controls have. For example, the required features will result in a form that reports an error for that control when no value is given during a submission.

The run method here actually performs the action. (For this example, I’m using something Catalyst-ish, but this could just as easily be used with CGI, CGI::Application, Plack, Jifty, or whatever you prefer.)

To use the login action in our web application, we’ll now need to send the form to the user. Here’s a Template::Declare template that should work:

template 'user/login' => sub {
    my ($self, $c) = @_;

    page_wrapper {
        form { { action is 'user/check_login', method is 'POST' }
            my $interface = Form::Factory->new_interface(HTML => {
                renderer => sub { outs_raw(@_) },
            });
            my $action = $interface->new_action('MyApp::Action::Login', { c => $c });
            $action->unstash('login');
            $action->render;
            $action->clear_messages;
            $action->stash('login');
            $action->render_control(button => { name => 'login' });
        };
    };
};

That would render an HTML form that the user can then fill in and submit. The new_interface call creates an instance of an interface class. These are responsible for connecting an object to the user interface. In this case, it’s an HTML form.

The new_action call then creates an instance of the action class associated with our interface class. Note that we can pass a hashref of arguments that will be passed to the constructor.

The unstash and stash calls are handy for remembering certain things about the form. This includes any error or success messages associated with processing the form. For example, if the user entered their login name, but the incorrect password. The processing bit (which we’ll see in a moment) would stash the error from trying to authenticate and would also stash the login name itself (but not the password since passwords controls don’t ask to stash their value). Then, when the form draws again, teh error message will be shown and the login name box will come prefilled with the previous entry.

The clear_messages call clears the messages from the action so that we can stash a clean slate for the action (we don’t want to see a message we already displayed again).

The render method renders all the controls associated with the action. (By passing the controls option you can output only a subset if you wish.) The render_control is used to output a button that is not directly associated with the action (in this case we just use it to get the user to send the form back to us).

Once the user clicks on our “Login” button, we’ll then want to process the input. That looks something like this:

sub check_user_login :Path('user/check_login') :Args(0) {
    my ($self, $c) = @_;

    my $interface = Form::Factory->new_interface('HTML');
    my $action = $interface->new_action('MyApp::Action::Login', { c => $c });
    $action->unstash('login');

    $action->consume_and_clean_and_check_and_process(
        request => $c->request->params,
    );

    if ($action->is_valid and $action->is_success) {
        $c->response->redirect('/index');
    }
    else {
        $action->stash('login');
        $c->response->redirect('/user/login');
    }
}

We get the interface and action objects same as before. We unstash the saved information again. Then, we do that long method, which is a shortcut for saying:

$action->consume( request => $c->request->params );
$action->clean;
$action->check;
$action->process;

The consume method takes the input and applies that to each control object.

The clean method then cleans up the values given to each control. In this case, for example, the login_name will have whitespace trimmed from the beginning and end since it uses the trim feature.

The check method then validates the inputs to determine whether or not we can reasonably expect the user’s input to have a chance at working. Here’s where most of the features added above will do their work. If the user does not enter anything for the username, the required feature will cause the control to be set as invalid.

Then, the process method will cause the action to process. Actually, the process method itself first checks to see if the input is valid and does nothing if it is not. If it is valid, it will copy the values from the control objects into the action attributes and run the run method defined on the action.

Finally, we can check the success or failure of the action and react accordingly. That’s how Form::Factory works in a nutshell.

Goals

Now that you’ve seen an example, here’s what I want to achieve in a nutshell:

  1. Actions should be well-structured, but very simple to build.
  2. Actions are tied to the user interface by controls, which are easy to extend.
  3. Common features of actions should be simple to define and associate with actions.
  4. User interfaces should be easy to build for hosting actions.

Actions are Well-Structured, Simple

Actions are just functors, function objects. At the center of each is a run method responsible for taking some action. To get there, however, you need input and after you get there you need output. Form::Factory uses Moose attributes to specify what input the action takes. Each has_control builds an attribute onto the action, but also associates more information about how it interacts with the user.

The output is a little less formally specified at this point, but comes out through a result object. The result object is a list of messages associated with the action along with some status information. It can also return a hash of other information for more general output. However, I’m assuming that most of the output of the action is the side-effects the action performs.

When an action is run, it goes through four basic phases to process:

  1. Consume. The user input is delivered to each control associated with each attribute on the action.
  2. Clean. The input in each control is filtered and corrected, allowing whitespace to be removed, numbers to be reformatted, etc.
  3. Check. The input of each control is validated and any errors discovered are reported. Processing stops here if there are errors.
  4. Process. The input in each control is copied into the action attributes directly and the action is run.

In addition to the features attached to the controls, you can specify general subroutines which will run during these phases. For example, if you have a form for changing a password, you might want a check that looks like this:

has_checker check_that_passwords_match => sub {
    my $self = shift;
    my $new_password = $self->controls->{new_password}->current_value;
    my $confirm_password = $self->controls->{confirm_password}->current_value;

    if ($new_password ne $confirm_password) {
        $self->result->is_valid(0);
        $self->result->error('the New Password and Confirm Passwords do not match');
    }
};

This allows you action’s run method to focus on the task it performs and nicely segments off all the checks. You can also logically group your checks and filters so that they are separate from each other. This makes it easier to subclass your actions or compose them from roles.

Controls are the Bridge

An attribute added to your action with has_control has a control object associated with it. This control does most of the work until right before processing. The control receives the initial input, the cleaned input, and is where the value sits when checked. It is not until it is checked that the value enters the attribute instance slot.

Without the control in place, i.e., if we assigned to the attribute directly, Moose would object to the input before we have a chance to do anything. I want to avoid exceptions in this case as well because input validation failures are not exceptional, they’re expected. We need a way to easily send back notes the user about corrections needed and would like to avoid spewing call stacks unnecessarily.

Controls also suggest how the user interface object should present them to the user. An interface is not bound to use these suggestion in exactly the same way as its peers. For example, a text control in HTML might just be a text input box while in a CLI interface might be presented as a command-line option that takes an argument. A check box, on the other hand, will be a check box input in HTML, but would be an option that takes no arguments in a CLI (it’s presence suggesting “on” and it’s absence suggesting “off”).

Features Modify Actions

A common feature of form handling is filtering and input verification. In Jifty these were called canonicalization and validation. Horrible. I’ve called them cleaning and checking. In general, though, the cleaning and checking you want to do is extremely repetitive. On almost everything I want to trim the whitespace and verify that something that is required is present. I often want to check input length for something too long or too short. Sometimes I want to check that email addresses or phone numbers are sane, etc. In Form::Factory, all of these are called “features.”

A feature is a generic object that modifies an action. These can be anything from a feature that is just used to flag the role for some reason to features that actually modify the structure and processing of the form. Features can also be attached to particular controls to clean or check an individual control’s input.

Interfaces Talk to the User

And interfaces are probably what make Form::Factory most unique. An interface looks at the controls for an object and presents them somehow to the user. Currently, I’ve written an HTML interface and a CLI interface. The HTML interface presents them using the matching HTML form controls. The CLI interface presents them as a usage snippet describing the command-line options the action accepts. You can define a single action class and use that action to define a web form or a command-line interface.

I plan to add a REST interface as well, which will be a variation of the HTML interface. Other interfaces I’ve considered (but have no plans for at this time) could include things like a Wx GUI, a PDF interface that builds a PDF with form fields and then consumes FDF documents created from that, an XForms interface, a XUL interface, or a curses interface.

Okay, so that’s my current contribution to the forms parsing arena. I’m still working the kinks out, so be careful if you give Form::Factory a try. Things will change. Otherwise, I’d love some feedback and help ironing out some of the wrinkles.

Cheers.

If you use Drupal and would like to automatically generate thumbnails of web sites, you might want to take a look at the Bluga WebThumbs module. Once that module is installed on Drupal and you have given the module your Bluga WebThumb API key, you can then inject a PHP snippet like this into a node using the PHP filter or into a theme file:

<?php
print bluga_webthumb('http://contentment.org/');
?>

Now, when you view the page containing that snippet, you will see a thumbnail generated using the Bluga WebThumb service within a few seconds that shows that web site. The thumbnail is cached locally on your server.

You can further customize your thumbnail with many options. You can read the full documentation for the module at Drupal’s web site as well. New features since March of last year also include a script for updating the thumbnail in place without requiring a page reload and a nicer looking (and CSS-styled) place holder, which is an improvement over the graphic previously used.

Last week, I was approached by the owner of Konigi.com to do some Drupal work. He’s a user of the Bluga WebThumbs module I wrote in March of last year and wanted to see it work with Drupal 6. I have finished the update and have made a full release of the Drupal 5 version of the module after making a few improvements and then ported the module to Drupal 6.

Cheers.

In the past, I’ve done a fair bit of work on Jifty with Jesse Vincent by way of my previous job. He had some Google Wave invites and I begged one off of him. After waiting a week, it came through and splat. Google Wave is a wasteland of nothing. Playing with some posts to myself led to about 10 minutes of use and then I hung up.

A colleague of mine, who (last I knew) does not have a Google Wave account had come across a LifeHacker post describe the “with:public” search that provides at least some content for Google Wave. However, that content is mostly just a dearth of “What can I do?” and “Anybody else here from Canada?” and “What happens if I post porn?” It was initially interesting, but I got about another 10 minutes and hung up again.

Okay, so there’s not much to do, but it’s brand spanking new. Thus, criticizing it now is like calling a bridge useless when the engineers have only just finished putting up the supports. There are couple things I’d like to note, though.

First, there’s not just a whole lot new here. Google Wave is merely a new combination of social networking, email, wiki, forums, chat, document and image sharing, and widgetry. If you’ve used a Wiki, Twitter, IM, and Email before, you have a good idea what Google Wave is already. They’ve just taken the next step and attempted to combine things in a way that will be challenging to scale, but I think Google is up to the task.

Second, that all said there is something I hope that can be achieved here, if not by Wave, by something like it. They consider Google Wave to be a reinvention of email. They have the idea that they might be able to replace email with it. Maybe. I think email has already been replaced by SMS, Twitter, and Facebook for many people. I end up using Facebook to send messages to folks much more often than Email these days and would probably use SMS if I weren’t so cheap and had someone other than my wife that I communicated with regularly via cell phone.

Problem: Services like Twitter and Facebook have a very significant problem. They’ve taken the Internet backwards by providing a single hosted service in the cloud. Email has an important advantage in that if my email provider has an outage, everyone else on the Internet is probably fine. If my email provider goes out of business or provides terrible service, I have the alternative to go somewhere else. Twitter and Facebook have competitors, but unless you can convince all your family and friends to move with you, you can’t leave them unless you’re willing to sever your ability to communicate with them.

Solution: Something like Google Wave or Twitter or a consortium of social networking sites needs to come up with a new decentralized mechanism for communicating between people. Whether that means you can switch services but they have hooks between one another to send messages and share friend lists or Google Wave provides some sort of decentralized platform for doing this independent of the social networking sites, something needs to happen here.

There are other dangers here to privacy and such that I haven’t even touched on either. Something like Google Wave (assuming Google Wave can become decentralized like email) should happen if social networking is going to continue to develop. That’s my thought anyway.

Cheers.

This morning I had an epiphany about a difference in project management style between the two major development jobs I’ve held. One style was like driving an empty bus and the other was like riding in a clown car. I am going to examine both as anecdotes from my perspective and try to avoid grandiose analysis.

The Empty Bus

So, I start the job and the first thing the company does is hand me the keys to the bus. Actually, they dither on what kind of bus to give me for several months before getting me a suitable one and give me a loaner to drive in the meantime. However, once on the bus driving, I am pretty much on my own. I have a destination to reach that has been vaguely described on a scribbled piece of paper. The directions are unclear and no one in the company has been there before. They keep changing the directions. But I get to drive. That is fun.

Every now and then, I pick somebody from the company up, they make changes to the directions and then they get off again before I go very far. Every six months, everybody climbs on to the bus and sits in the very back. They do a lot of yelling while I park and then they take away my scribbled directions and give me new scribbles to follow and a new destination to reach. But I get to drive. That’s usually fun.

All in all, I am asked to develop software with very little cooperation or direction. I am left on my own to make almost all the decisions. Even though I have weekly meetings with my manager, I am not really given much feedback on whether I’m going the right way. He’s not a developer, so he doesn’t really know enough about what I do to give me useful feedback. My quarterly reviews aren’t very cooperative or helpful, they are more about the manager wishing I would drive faster and make fewer mistakes (mutually exclusive goals when you think about it).

I nearly get into a wreck a couple times, but there’s no one on the bus to help me out. Don’t get me wrong, I’m not an excellent driver. I’m still learning, but some help should help things go faster in process, you’d think. Usually, though, my directions are so unclear and difficult to follow that I am directed to get into wrecks. This is not actually all that fun as time goes on.

The Clown Car

Clown cars are funny. They drive around in circles and then the doors pop open and an absurd number of hilarious characters hop out of the little car. This job is not quite like this. It’s actually more like a really crowded minivan, like the trip I took the other day with my wife, son, dad, mom, brother, sister, and her boyfriend, all crammed into our little Pontiac Montana. But now, imagine, that all of these people have a stake in where the van goes and have a slightly role and different idea about how to go about getting there. Now, we’ve got a good analogy. Clown cars are fun, though.

The CEO’s seat is next to mine and he gets in and out of the van whenever he feels the need. He’s a busy man: lots of vans to help. Usually, he gets in right before we wreck or near the major turns to make sure we drive carefully at those point and turn the right direction. Directly behind me sits an analyst whose job it is to navigate. He tells me where to go and annotates those instructions pretty regularly. Beside him sits another analyst whose job it is to talk to the customer and figure out what they want. He then tells the first analyst and me where we need to think about going next. I get to drive, though, they keep reaching from the back for the wheel and the pedals. That’s annoying, but still fun.

Behind the analysts sit a whole team of project managers, executives, sales people and between them and between the front seats sit a bunch of other engineers. Sometimes the other engineers help drive, make suggestions, and they often critique or commend my driving. There are a lot of people in this minivan, sometimes there’s a lot of yelling about what to do next. All the activity does make for quite a bit of fun, even if it gets a bit distracting at times.

I get to drive. As I mentioned, sometimes the analysts and engineers have their hand on the wheel and help push the pedals for me. This is pretty fun too, unlike the car though, this actually gives us a lot more control. We seem to be getting places in a much more controlled way, though we do have to control our speed much more carefully. It might take us longer, but the drive is fun along the way.

Software development in this style takes away some of my freedom as a coder. That’s a bummer in some ways, not as much fun. I like control. Yet, it also lets me focus on my strengths while others worry about talking to customers, making sure we have a plan that does what the customer wants, and while the constant feedback makes it hard to see the big picture, I usually have a lot of warning before I drive off the road or get into a wreck. Overall, this is more fun.

So far, I prefer the “clown car”/stuffed minivan to the empty bus. It’s less bipolar and more slow, steady, and directed.

Cheers.

Last week, a coworker let me know about this. It’s been on Slashdot and such, but I’m a little behind on my feeds. For some reason, the competition intrigued me so I’ve put together a set of Perl classes that will allow agents written in Perl to control the Mario AI simulator using the server agent. I considered trying Java, but I started and quickly remembered why I hate Java, so I quit that and wrote this up over the weekend instead. You can fork it and give it a look here.

It includes a very stupid and blind robot for testing at this point. However, it is able to get through most levels I’ve tried on difficulty 0 (though, not many on any higher difficulty). Once you install that and fetch the latest JAR file from the Mario AI competition site:

You can then run an agent via:

java -jar iMario.jar -server on

to start the server, then:

cd AI-Mario
bin/agent.pl

This will run the blind Perl agent (named AI::Mario::Agent::Simple) on the same level over and over. A more interesting run might be:

bin/agent.pl --config AI::Mario::Config::Random -o level_difficulty=2

This will run the same agent through a random series of levels (restarting each time the agent wins or loses) and sets the level difficulty to 2—which means, the simple agent nearly always loses.

The system is built with Modern Perl (i.e. Moose) and uses POE and POE::Declarative to talk to the Mario AI server agent. You can implement new configuration (to customize how options are set at the start of each run) and agent and set them to run using the arguments available on the CLI.

To write an agent, for example, you just need to create a class that does the AI::Mario::Agent role and implements the required methods.

  • name: should return the agent name to be sent to the server agent as greetings
  • reset: a method called to reset the agent when a simulation is about to start
  • update: a method called during each frame, given an observation object (see AI::Mario::Observation) and should set the left, right, duck, jump, and run controls appropriately to keep moving.
  • fitness: this is called at the end of the simulation with a fitness report to tell the agent how well it did (see AI::Mario::Fitness)

Then, you can run your agent via:

bin/agent.pl --agent MyAgentClass -p foo=1

where you can pass a series of parameters to the agent constructor using the -p option.

I plan to add a bit more to it, but the interface is mostly set.

I’m not even sure they’ll accept a Perl submission, but it seems to be implied by the instructions on the site. I’ve asked to be sure. If I can confirm that, I’ll also add a Makefile.PL and such so that the solution can be packaged with all the required libraries to make it run.

I still haven’t decided whether or not to enter or just goof around, but whatever. :)

Cheers.

Update: Julian already replied. Yes, Perl submissions would be accepted. I’ll definitely try to add a submission script which will help build all the requirements to submit.

I was goofing around with HTML 5 canvas for giggles this week. The canvas allows you to draw using JavaScript to a section of the page. HTML 5 defines a “2d” context, which allows you to draw lines, polygons, images, boxes, etc. You can style your stroke, fill, and shadow, etc. It’s pretty easy to goof around with. Oh, and if you’re using Internet Explorer, these won’t work. You’re out of luck because Microsoft is too cool for canvas.

Box Fractal Renderer

First, I built a box fractal renderer, which are my favorite things to doodle on graph paper.

Fractal Coastline

Then, I played some more with fractals and eventually ended up with this fractal coastline builder. Each view of this will be a little different. Some look more like coastlines than others.

Anyway, just some random goofing off I thought I’d share.

Cheers.

From the perspective of myself, a software developer, time estimation always sucks. I believe I’m working in an environment which has a better view of time estimation than those I’ve worked in before, but we still do it and I think that it’s sucking is inherent. I’ll explain why in a minute.

However, I want to say that time estimation is probably essential for project managers, product managers, executives, sales, and the like. For them, perhaps they can provide a barometer giving them a good indication of where things will be in the future. However, I don’t speak for any of them, I’m speaking from the selfish position in which I stand (as we all do and would admit when we’re being honest).

Time Estimates are Guesses

A time estimate is, by definition, going to be inaccurate. Until someone finds a prophet willing to use his gift for mundane predictions like, “How long will this new feature take to implement?” Or, “What’s the timeline on this fix?” The answer will be wrong.

To a person without the training it may seem like a software developer works magic and comes up with solutions, but that’s not the case. She’s merely working through developing an explicit set of directions, logic, and math to allow the computer to do something. However, the complexity of a problem is rarely well-known until you try to work through it. Thus, you don’t know what problems you are going to run into.

This is further complicated by the fact that you have to work within some framework including programming language, available libraries, code already in built up, customer expectations, code maintainability, solution flexibility, etc. A software developer often has to make decisions and compromises along the way and it’s not always clear what the ramifications of each of these decisions will have.

Implied Margin of Error

All time estimates include an implied, but rarely recorded or even knowable, margin of error. When I estimate the time it will take to do something, I will often say something like, “Well, since I just did a very similar process in importing that fail for client X, it will take about 2/3rd’s as much time to do it for client Y since I’m familiar with the code and added some pieces there to make doing this task easier.” I’m implying a narrow margin of error.

On the other hand, I might say, “Well, the code in that part of the application was not well future-proofed and this will build on there. I may have to do some refactorings, so I’d say we’re looking at 2 days, plus or minus a week.” I’m even stating it here, the large margin error.

Yet, what if the first task turns out to take 5 times as long as I said, does that mean my margin of error was wrong or does that mean I was operating on incomplete information?

When a meteorologist says, “50%” chance of rain, is he wrong when it rains on your house? Or when it doesn’t? No, it was just an estimate, which is a statistical entity that cannot be analyzed on those terms as a data point. Only after you have a collection of data points that shows that when the meteorologist suggested 50% chance of rain, it actually rained 60% of the time. Then, he’s been wrong (and even in this analysis, you have to be careful about data bias and all the other statistical issues your analysis may contain).

Large Estimates

Getting into the actual business case, we find that making a large time estimate on something will get you into trouble when you make the estimate.

I recall a case when I suggested I was starting on something during the last week of December after Christmas. I broke the task down into all the steps I would have to do to complete the task and concluded that an reasonable estimate of time would be the end of the second week of February. I told this to the executive in charge of the product and he said, “We’d really like to be able to tell the clients the start of the third week in January.” I said, “No. I don’t think that’s reasonable.” He said, “We’d really like to be able to tell the clients the start of the third week in January.” In other words, “I don’t care what your estimate is, this is the estimate we’ll use.” My response, “I will give you something at that time, but it won’t be finished.”

In this case, it actually took all the way through April to get it done, largely because the project ended up having twice as many features as originally spec’d.

As a software developer, I am strongly motivated to report short estimates to avoid pressure from above. Sometimes, however, a long estimate is the only one that will work, but I always expect to be criticized if I give a long estimate on something.

Short Estimates

If I end up giving an estimate for a project that actually takes me 2 to 5 times as long to complete, then I’m in big trouble. These are the times when executives yell at people. Directors and project leads ask you what lessons you’ve learned, and if someone really gets upset, you have weekly phone calls with someone in HR listening in.

So here’s where making a long estimate for a short task seems like a good idea. But then, what happened in the last section? There’s no win in this game.

Accurate Estimates

So what happens when you get it dead on? Do you get a big promotion, cigars and cognac with the executives, and a fat bonus? Nope. You just plod on. You might get such if you made a long estimate, got away with it, and then delivered way early, but that might just mean you’re a good con-artist, not necessarily good at estimating your time.

What’s the Answer?

I don’t have one for managers. That’s not my problem, but managing my managers is. So, here’s the solution (at least what seems to work best for me at this point):

  1. Communicate frequently with the bosses. Anyone you know to be a stakeholder in your project should get something from you often enough to keep them in the loop, but not so often they get annoyed. Hard to balance, but in generally you ought to be talking to your supervisor at least once every day, your project/product managers should get information at least twice a week and as much as once every day, and executives should get direct feedback from you once a week if they have a stake.

    And if you find out something is really critical to someone, update them as often as your supervisor. If it’s an emergency or something critical to a sales pitch being made tomorrow, send out an update at every stopping point.

    Regular status reports help managers feel in control and you want them to feel in control. If they feel out of control, expect them to blame you for not telling them what was going on.

  2. Make your guesses as good as you can. On complex tasks, break the task down as far as you can ahead of time and estimate each piece. Give yourself an explicit margin of error and percentage of accuracy you’d place on that margin. Sum up the time estimates and margins and use that to present you estimate. Take some time to do this well, don’t rush and spend more time considering the harder to estimate parts.

    When you give your estimate, you might share some or all of this with your supervisor (depending on how well you think he’ll be able to cope with the information). Don’t give this to anyone else, but use this information to determine how to talk about it when reporting on your estimate. Use vague, fuzzy language for parts you aren’t sure about and use concise, direct language on the parts you are pretty sure about.

  3. Stand by your estimate. If your boss says, “Unacceptable,” or doubts you, listen to them and explain more about your estimate. Especially if you’ve worked through the time it will take in details, wavering in a moment of doubt is probably not going to serve you or anyone.

  4. Be flexible. If the bosses need a shorter estimate for some reason or even expected a longer one, use that to your advantage. If you’ve done your homework and got it with you, perhaps you can negotiate parts of the spec away that aren’t as critical in order to cut down on time. Or perhaps there are things you can iron out a little better and get working really well in the time you have. Or perhaps you can use that opportunity at the tail end to get better testing and quality in place.

  5. Meekness wins when all else fails. Never ever burn your bridges. When it hits the fan because you prove to be the woeful prophet you are, stand by your mistake, but stay calm. Be humble. You might have a really good idea what went wrong, but your bosses are pointing at something else. Point that out, but don’t beat them with it if they disagree.

Other than that, be you. I work really hard to never blame someone for a mistake, even if they made it. I do not tell on fellow employees when they aren’t doing their job. However, I try to make it a point to point out excellence in my fellows whenever I can. I don’t feel like it’s my job to do management (which, in my mind, means correcting and rebuking employees when they fail at something), but pointing out excellence and success is everyone’s business. If I really appreciate something a fellow developer has done or someone on the business side writing a really great spec, I try to make it a point to say so in conversation. Unfortunately, being the loner that I am, I don’t think I notice these things as often as I could.

Time estimation sucks, but it’s pretty much inevitable. The key is understanding why it sucks and how to deal with that.

Cheers.

Disclaimer: IANAD: I AM NOT A DOCTOR. Please consult one prior to engaging in any attempt at weight loss. I provide no recommendation you do things my way and take no responsibility for your health.

Disclaimer 2: I am not affiliated with, have never given money to, and have no relationship at all with the Unnamed Diet System I have based my diet plan upon. I do not recommend for or against their system, I am just sharing the fact that I my own variation of it and have had success.

Since my birthday this year, I’ve been working to lose a little extra weight. I now weigh around what I did when I was in high school and am still losing. I did this to show solidarity with my wife who wanted to lose a bit, but I needed to do this for myself as well.

Wanting to lose weight presented a few problems for me. First, I can not stand to do what somebody else tells me to do. My wife calls that being “obstinate-defiant.” Depending on my mood I usually either say I’m just selfish or I have a built-in distrust of the crowd. In any case, it was clear that unless I could tweak my plan a bit, I wasn’t going to be be happy with it. I’ve ended up tweaking less than I thought I would, but I still can, so there.

My next problem is that I refuse to do cruel or unusual dieting. I’m not going in for any fad or diet that drastically changes things. I’m not giving up cookies or eating grapefruit or doing Adkins. Regardless of what science there is or is not backing these things up, anything that changes what I eat is going to make me grumpy. I love food and there is no shame in that. The shame comes in consistently eating more than I need. I also find the idea of using a pill or surgery repulsive (no offense to those who do such, I won’t). For me, this process is about developing self-control, which means I need to learn to do it and my wife provides enough accountability to that end.

While metabolism and other factors adjust a person’s dietary needs, failure to consume enough calories to maintain a person’s current weight will cause a reduction. (Unless, which I suppose is possible, a person’s body is somehow capable of storing fat, but incapable of using it. I don’t know if any such disease exists, but I don’t have it if it does, so it’s not my problem.) Therefore, my diet plan would have to be something as mundane as journaling what I eat.

Which brings up my next problem, counting calories is too easy and not really addressing the full magnitude of the problem. Because not only should I eat less, but I should encourage myself to eat healthier. The system ought to take other factors into account.

My final problem is that whatever it is I do must be something I can do on a computer. I sit in front of one for around 8-12 hours per day. I can set my computer to remind of things, I can share things between myself and my wife to provide accountability on my computer, and while I like writing down notes, particularly when I’m brainstorming, I really don’t want to do all the math we’re talking about in my head all the time. It’s too tedious.

Fortunately, my wife previously went on and successfully completed a plan using Unnamed Diet System for which you may have seen ads. This system met all my basic requirements. Yet, other than they way they count points, they don’t provide any value to me, at least none I would pay for. Fortunately, everything I needed was published at various places on the Internet and I built myself a Google Docs spreadsheet to do it.

I provide a link to a version of it here for anyone interested in weight loss on similar terms. Go re-read the disclaimers again now. I’ll wait… Done? Okay, I don’t recommend this plan to you, but if you find the spreadsheet useful, great. I’m providing it under a Creative Commons 3.0 license.

If you have a Google account, you can create a copy of the spreadsheet to use it or you can download it in another format to use with Excel or OpenOffice (I think, haven’t tried that).

To use it, I first scroll right until I fill in some information about myself. This sets up the basic tolerances for my diet plan based upon sex, age, current weight, height, daily activity, etc. Then scroll back left and log my consumption. Under each meal, the wide column is for a description of the item eaten and the narrow for recording the points. Once I have “0” points left for the day, I stop eating. The spreadsheet does have a weekly allowance of extra points that I can use as well to indulge in something or just allow me to consume all the points for a day without worrying about going over by a couple. I use all the points I have for a day unless I’m really not hungry. This is not a starvation diet, so I try to use up as many of my daily points as possible. I do not worry too much about using or not using the weekly points. I often consume most of them.

The formula for calculating this points is simple, but elegant in that it encourages me to get more fiber and avoid fatty foods, while consuming fewer calories than I need to maintain my weight:

Points = Calories / 50 + Fat (g) / 12 + MIN(Fiber (g) / 5, 1)

I’ve been told that Unnamed Diet System actually divides Fiber by 4, but whatever. I have embedded a couple calculators in the spreadsheet for the times when I’m too lazy to do the math in my head.

I weigh myself once a week to track my progress (on a separate spreadsheet).Every 10 pounds, I adjust the chart to the right since the spreadsheet gives one less point per day for each 10 pounds I lose. I also copy the spreadsheet (actually, Terri manages this part) each week and blank it out to use the next week.

Eventually, I should reach my goal weight (I haven’t really decided what that is). When that happens, I’ll need to adjust the spreadsheet to deal with maintenance. When that happens, I will give myself more points until my weight stabilizes. I plan to continue recording points for the foreseeable future this way.

Cheers.

I just finished writing a test that discovered I’d made a rather dumb and (upon looking back) rather obvious mistake in a return value in this Perl application I’m working on. The mistake involves a certain errant combination of return and and must be dealt with carefully.

The particular bit of code looks something like this:

sub blah {
    return $foo and $bar;
}

For those who don’t know Perl intimately, Perl has two “and” operators. One named “and” and the other named “&&” like C. These are not strict synonyms. They are both shortcut operators, but the “&&” and the “and” sit at very different places in the operator precedent order. In Perl “&&” has a relatively high precedent and “and” is very, very low.

Back to the problem: this return was returning true when $foo was true and $bar was false. After rereading this line I smacked my forehead and said, “Duh!” The problem is that “return” actually has a higher operator precedent than “and” so this is how Perl would break it out if it showed the AST with parenthesis:

sub blah {
    (return $foo) and ($bar);
}

This means the code immediately return $foo in all circumstances. I might as well have just written:

sub blah {
    return $foo;
}

The solution, then, is to either use the “&&” operator:

sub blah {
    return $foo && $bar;
}

Or use explicit parenthesis:

sub blah {
    return ($foo and $bar);
}

Or don’t use the “return” operator (since subroutines in Perl always return the value of the last expression executed):

sub blah {
    $foo and $bar;
}

Cheers.