Developing RESTful Web Services in Perl, Part 2

Part 1 of this article was posted on onlamp.com. Due to changes in the format of O’Reilly’s online publications, they are no longer interested in Part 2, so I have published it here. Sorry for the delay but it took some time to contact the editor to learn all this.

In my previous article, I described the process of building a RESTful web API from the server-side. This article will describe how we write the client to the web services server provided in the first article.

The source code for the client and server described in this article is available by saving the following links:

• Client: book
• Server: library.cgi

There are some instructions on how to install them in comments in each file, but you’re mostly on your own. I can say that they should work (do work for me) if installed correctly and if the required libraries are in your library path. Sorry that I haven’t made these easier to use.

RESTful Client

The RESTful server written in the previous article helps me track the books in my library. This time, we’ll look at a command-line client that takes advantage of the API. The principles applied to build this client could be used to in your web application to access web services APIs provided by third parties or to build a GUI application or anything else. A command-line just provides a good simple interface with which to demonstrate.

The command-line script is named book and operates through the use of subcommands. For example, to list all the books in the system, you can run:

./book list

In cases where a new book is added or updated, you specify the update using a file name.

./book update 1-56563-833-6 treasury.yml

Without further ado, let’s jump into the code.

Setting up the Client

Before getting into the commands themselves, there is some setup performed by the script. First, we have a constant named HOST. This constant is just the URL of the REST API server.

I’ve also defined to additional helper functions that will show up throughout this article. The first, barf(), is called whenever an error occurs. Since our server returns errors as HTML, it just pulls title of the error from the HTML header and the rest of the error message from the first paragraph. The second helper function is slurp(), which is really nothing more complicated than the function by the same name in File::Slurp (but without an additional dependency, which I was attempting to avoid for the simple demo). All this does is suck in all the text of a named file and return it as a scalar.

The last bit of setup is that I’ve initialized one global variable named $ua for convenience. This variable contains a freshly initialized LWP::UserAgent object. The only library from CPAN this program depends upon is libwww-perl. I’ve also imported some helpers from HTTP::Request::Common, but we’ll get to that in a bit. Now, let’s talk about the interesting stuff.

Listing Resources

The first and simplest command we can run is to simply check to see what resources are available. You can see all the books stored by listing them using this command:

./book list

Calling this command causes the script to run this bit of code:

# GET /=/model/book
my $response = $ua->request(GET HOST.'/=/model/book/id');

# On success, find the link and print them out
if ($response->is_success) {
    my @links = $response->content =~ /\bhref="([^"]+)"/gm;
    for my $url (@links) {
        my ($id) = $url =~ /([\d-]+)$/;
        print "$id: $url\n";
    }
}

# On failure, barf
else {
    barf $response;
}

Our first act is to grab a list of books from /=/model/book/id. As we saw in the previous article, this should return an HTML file containing a list of links as part of the API. If the request was successful, we pull all the links out using a regex and print them out to the end-user. If the request fails, we barf.

The key bit of code here is all within the call to the GET subroutine, which is exported by HTTP::Request::Common. This performs all the extra work we need to build a standard GET request. Another alternative is to use the get() method of LWP::UserAgent, which would look very similar:

my $response = $ua->get(HOST.'/=/model/book/id');

Moving on, I hope I haven’t turned your stomach with how cheaply I’ve parsed the URLs and IDs out. This is not an ideal use of regex, but it does the job. If I were to do this the right way, I ought to use a decent HTML parser (I’m partial to HTML::TreeBuilder) to pull the links out more carefully. Another solution would be to alter the server so that it returned the links as a YAML file and then parse apart the YAML data to get the information I want. I’m lazy and wanted to avoid additional module dependencies at all cost on the demo, so I do it this way.

This will return a list of IDs, which may be ISBNs or just a number, which isn’t terribly useful. A simple enhancement to the system would be to change this to use the “title” field or something else, but we’d also have to enhance the server to support that.

Reading a Resource

Once we know what resources are available by the list, we may then want to check one out to get more information about it. We can do this using the read command. For example,

./book read 1-56563-833-6

As you will recall from the server, we can read data using a GET request to /=/model/book/id/<ID>. This is what we do when the read subcommand is executed:

my $id = shift @ARGV;

# GET /=/model/book/id/[id]
my $response = $ua->request(GET HOST.'/=/model/book/id/'.$id);

# On success, print the file
if ($response->is_success) {
    print $response->content;
}

# On failure, barf
else {
    barf $response;
}

We grab the ID given from the command line and use LWP::UserAgent to GET the book resource we’re interested in. On success, we then print the file out and on failure we barf.

If we wanted to do something more interesting with the resource file like present in a form or place it into a table or something, we’d use a YAML parser to decode the data and then manipulate it.

Creating a Resource

Now that we’ve looked at the resources in our (presumably empty) database, let’s look at how we added them. The first command to consider is, of course, for creating a new book record.

./book create treasury.yml

The command slurps up the file you name and uses it to POST to /=/model/book to create a new book record. We then scan the response to make sure we know what ID was assigned (because, if you’ll remember, not all books have ISBNs, so the server provides an alternate identifier in such cases).

my $file = shift @ARGV;

# Slurp up the contents of the given filename
my $book_data = slurp $file;

# POST /=/model/book
my $response = $ua->request(POST HOST.'/=/model/book',
    'Content-Type' => 'text/yaml',
    Content        => $book_data
);

# On success, return the new ID assigned to the resource
if ($response->is_success) {
    my $url = $response->header('Location');
    my ($id) = $url =~ /([\d-]+)$/;
    print "$id: $url\n";
}

# On failure, barf
else {
    barf $response;
}

As you can see, this is exactly what we’ve done. Since the server requires the data to be specified as text/yaml, we’ve made sure to note that here. If the file is improperly formatted, the server will take note and return an error status, so I’m not being too picky about making sure the data is clean. However, you might want to make sure your data is sane before sending it in such cases just to avoid potential problems.

The other important detail to notice here is how we get the ID assigned to the resource back. We do it by checking the Location header. This is because we’ve build the server to return a 201 Created status with a Location header referring us to the new resource location. If we were to perform an immediate GET on that location, we’d get the resource record we just saved back from the server. In fact, that’s probably what I should do. Instead, I’ve just ripped the ID off of the URL since I know what it will look like. This is a minor breach of the opacity principle of URIs, so if you’re sensitive about such things, I recommend you take the extra step and pull the ID from the returned YAML data.

Updating a Resource

Once we’ve added a resource, we might want to update to correct a typo or submit additional information to the record. We can do this by running:

./book update 1-56563-833-6 treasury-updated.yml

As with the create command, we slurp up the file argument, but instead of a POST, we PUT. We also use the URL included the given ID.

my $id   = shift @ARGV;
my $file = shift @ARGV;

# Slurp up the given file name
my $book_data = slurp $file;

# PUT /=/model/book/id/[id]
my $response = $ua->request(PUT HOST.'/=/model/book/id/'.$id,
    'Content-Type' => 'text/yaml',
    Content        => $book_data,
);

# On success, just announce success
if ($response->is_success) {
    print "Updated $id\n";
}

# On failure, barf
else {
    barf $response;
}

We again make sure to let the server know we’re passing it a YAML data file. Here the returned result is pretty empty, so we ignore everything but the status code and just return a success message. If we had implemented resource renaming on the server using PUT, we would probably need to watch the response for the new resource URI to make sure we fetch the new ID. Since we didn’t, we don’t.

Deleting a Resource

Finally, when a book gets lost or sold or just thrown away, we can delete it from the library. From the command line, this looks like this:

./book delete 1-56563-833-6

Internally, the command is again very simple. This time, we just need to post a DELETE to the appropriate URL, /=/model/book/id/<ID> and check to see if it succeeded or not.

my $id = shift @ARGV;

# DELETE /=/model/book/id/[id]
my $response = $ua->request(
    HTTP::Request->new( DELETE => HOST.'/=/model/book/id/'.$id )
);

# On success, announce it
if ($response->is_success) {
    print "Deleted $id\n";
}

# On failure, barf
else {
    barf $response;
}

First thing to note is that here we build the DELETE request ourselves. This is because HTTP::Request::Common did not provide a shortcut for DELETE when I originally wrote this client. I am happy to say that such a method exists as of 5.814 of libwww-perl. In case you aren’t able to use the latest version for some reason, this is how you live without. Fortunately, it’s not difficult to do this on our own.

Other than that, this should look remarkably similar to the update command minus the slurpy file action.

RESTful Resources

Now that you know the basics of building a REST server and client, you’re ready to move on to bigger and better things. Here are some resources for learning more about REST, for creating and using REST interfaces without building one yourself, for enhancing your REST interfaces, and some existing REST interfaces you might want to model or take advantage of.

RESTful Documentation

If you’re going to build a REST interface or work with one, it is helpful to find some details on what such things are commonly involved with. The most commonly referred to reference on the subject is the REST Wiki. If you want to know the general outline of how some guys that think a lot about REST APIs in the abstract, this is a good resource.

An even more vital resource is the actual HTTP standard. It describes what the various request types and response types are for and how user agents should expect to deal with them. Since REST is tightly bound to HTTP, sticking to the proper behavior in HTTP is very important. Therefore, I recommend becoming familiar with RFC 2616, which helps define the HTTP 1.1 protocol itself.

RESTful Tools

If you don’t want to mess so much with the code and just build on the foundation already laid by someone else, the best tool for the job that I currently know of is OpenResty. This is a stand-alone REST database system. It’s essentially a middleware platform for accessing a database over HTTP and is a pretty good system to emulate.

I’m also partial to the REST plugin provided by Jifty. Just by implementing a few models and actions with Jifty, you get a REST interface to them for free. See Jifty::Plugin::REST for the server side implementation and Net::Jifty for the client.

Both of these tools have a very similar feel to the implementation I’ve built for these articles because I like the style of Jifty’s implementation.

RESTful Extensions

The most significant RESTful API add-on I know of is OAuth. This tool provides a standard mechanism for sharing protected data between disparate services via REST. For example, let’s say you’re building an application that automatically sets the profile photo for several different social networking sites. You don’t want to store these photos, you just want to grab them and update a bunch of social networking sites. You could use OAuth to allow your users to grant you permission to access their Flickr or Picasaweb accounts without asking them for their username and password, which is one of those things that serious privacy advocates go bonkers over.

There are some big players implementing this and I’d love to see more mashups using this rather than asking me for my username/password to get photos or load contacts.

RESTful Services

There are lots and lots of RESTful web services available. However, I will highlight just a few that I have worked with personally in the past.

The first two are by Amazon. Amazon’s Web Services are all REST based, but the one’s I’ve worked with include S3 and EC2. With Amazon S3 you can store files on their services and be charged in micro-payments for just the storage you use and the amount of data transfer (which is pennies per Gigabyte). With Amazon EC2 you can run Linux-based servers that are started and stopped and manipulated using a REST-based service. These are again paid for in micro-payments by the hour (starting around 10 cents / server hour last I knew). There are Perl libraries already available for manipulating both of these without having to know the API directly too (search CPAN for Amazon).

The other I have worked a little bit with is Intuit Quickbase. Quickbase allows you to build database applications with a point-and-click interface. You can then pull and push data in and out of the system using a RESTful interface.

The last one I want to mention is Hiveminder. Hiveminder is a web site for managing your to-do list. Hiveminder provides a number of different interfaces including an IMAP server interface and a RESTful Web API. Hiveminder is built with Jifty and has a special sub-class of Net::Jifty available for accessing the web API, Net::Hiverminder.

Conclusion

There’s more that could be said and more resources I’d like to refer, but I think this article is plenty long. Adding a RESTful interface to a web application is a relatively simple thing to do and is great for giving folks a clean way to access and manipulate your application’s data.

Please feel free to comment here and I will try to answer any other questions or comments.

Cheers.