Form Data

One of the first major revisions to HTML (HTML 2.0) focused heavily on the addition of forms to the html cannon, with form and input tags being the major addition. Forms allow clients to send data with their request as part of the request’s body - utilizing HTTP and its underlying foundations to serialize this data as a series of data packets, much as the web servers we create do in sending web pages and other files to the browser.

Let’s see this process in action by adding a form to our Gallery to allow the user to change the gallery’s title client-side. To help keep our server code clean, let’s add a function, generateTitleForm() that will produce and return our form as a string that can be concatenated into our main gallery page.

function generateTitleForm() { return '<form action="/">' + ' <input type="text" name="title" />' + ' <input type="submit" value="Save Changes"/>' '</form>'; }

Then, we just need to add a call to this function to where we are building our HTML in our serveGallery() function. If we launch the server, type something into the text box, and then sumbit the form we’ll get a 404 error. If we look at the url in our browser, we can see why - it’s added a query string (the part after the ?) which is a sequence of key/value pairs. In my case, this was: http://localhost/?title=My+Best+Gallery+Ever%21. As our server is currently only matching “/” without a query string, it misinterprets the form data as a new address.

A better approach would be to strip the query string (which is not part of the resource request) before we start looking at what resource was requested. We can do so by splitting the url string - i.e.:

var resource = req.url.split("?")[0];

Now our gallery page will once again load, but we can go one step further and capture and parse our query string into a JavaScript object (basically, a key-value map) using the querystring module:

var querystring = require('querystring'); var query = querystring.parse(req.url.split('?')[1]);

The resulting query object is a associative array of key/value pairs. Because the query string will probably be used differently by different pages, we need to provide it to them. We could do this by adding a variable to our various serve() methods, i.e.:

function serveGallery(req, res, query) {...}

But this could potentially force us to change a lot of code. Remember, JavaScript is a dynamic langauge, which means we can alter the structure of objects at any time, including our incomingMessage instance (our req). Since the querystring is part of the request, it makes a certain amount of sense to attach the parsed query object to it:

req.query = query;

If we do this in the request handling function, the query object will be available to all our serve() functions.

Changing our Title

With the query object we can now save our new title and use it to generate the title in the Gallery page. We'll use the ?: ternary operator to make sure we have a title to set, first.

function serveGallery(req, res) { var title = req.query.title ? req.query.title : “Image Gallery” // Load our images fs.readdir('images', function(err, files) { if(err) { console.error("Error in serveGallery", err); res.writeHead(500, {'content-type': 'text/html'}); res.end('Server Error'); return; } // Create a list of image tags from the directory contents var imageTags = files.filter( function(file){ // Filter files for images return fs.statSync('images/' + file).isFile(); }).map( function(file) { // Wrap file with img tag return "<img src='/images/" + file + "'/>"; }).join("\n"); // Show a gallery page res.writeHead(200, {'content-type': 'text/html'}); res.end( '<!doctype html>\n' + '<html>\n' + ' <head>\n' + ' <title>' + title + '</title>\n' + ' <link href="gallery.css" type="text/css" rel="stylesheet"/>' + ' </head>\n' + ' <body>\n' + ' <h1>' + title + '</h1>\n' + generateTitleForm() + ' <div class="gallery">' + imageTags + ' </div>' + ' </body>\n' + '</html>\n' ); }); }

If we load our server now, load our gallery, and submit a new gallery name, when the page reloads, our new name is proudly displayed. But if we just type http://localhost into our address bar, our title goes away, and we’re back to our default. Why?

Remember, JavaScript has functional scope - so the title variable we have declared in serveGallery() only exists within that invocation of serveGallery() - the next time we serve the page, it’s gone. We can share the variable across invocations by moving its declaration up in scope - in this case, by making it global within our server script:

var title = “Image Gallery”; // default name

Then, we need to be sure that we’re using that variable and not declaring a new one within serveGallery()’s scope:

function serveGallery(req, res) { if(req.query.title) {title = query.title}; ... }

We can use req.query.title to test if the user entered a title. Our concern is that title is undefined, in which case we don't want a gallery title to be undefined.

Now, as long as our server is running, our gallery title will be the last one we submitted (or the default, if we didn’t submit one). But if we shut down the server and restart it... we’re back at the default.

Persisting the Title

What we need is what computer scientists refer to as persistence - keeping data around between the individual uses of an application. You’re already intimately familiar with this concept, because every time you write code in a text editor, you save your code as a file (hopefully), before you close the editor. This is file-based persistence, and it would be a good fit for us as well.

What we want to be able to do is save any changes to our title to a file, and load those settings when we start our server up again. Such a file is often referred to as a configuration file for obvious reasons - it configures our application when we start it.

We’ll use JavaScript Object Notation (JSON) as the format of our file. This is a text-based serialization format that looks very much like a JavaScript literal object definition:

{ “title” : “Image Gallery”, “author”: “Me” }

The biggest difference is the keys must explicitly be string literals, unlike when using a literal object definition, where we can leave off the double quotes. We’ll save this file as config.json in the top level of our web application folder.

We’ll also want to load the configurations from this file when the server starts up. Here we can actually use synchronous file access, as we really don’t want the server to start serving web pages until it is fully configured:

var configJsonString =fs.readFileSync("config.json", {"encoding":"utf-8"}); var config = JSON.parse(configJSON); var title = config["title"];

One of the nicest parts about using JSON is that it is natively supported in JavaScript. We just use the JSON.parse() function to parse our string - and it gives us back a JavaScript object containing exactly the keys and values from the JSON. We can also convert any JavaScript object to a JSON string with JSON.stringify() (beware, there are some challenges with cyclic references that we’ll address later).

We also want to write the title back to this configuration file when it is set. Let’s place this into its own function:

function saveConfig() { var data = JSON.stringify(config); fs.writeFile("config.json", data, function(err){ console.error("Error saving configuration", err) }); }

Here we will want to use asynchronous functions, and we get to utilize the JSON.stringify() method to turn our configuration object back into a JSON string, which we’ll write to the file. If the function encounters an error, we’ll write it to the console so we can review it.

We’ll also need to call this function when we have a new title, so we’ll change our serveGallery() function slightly:

function serveGallery(req, res) { if(req.query.title) { title = query.title; config.title = query.title; saveConfig(); } ... }

Now our gallery’s name is retained, even when we restart the server.

Working with Post Data

While the querystring approach certainly works, it does leave something to be desired - namely, not having a querystring appear in the url. Moreover, the query string itself is limited in length, so for long forms or forms with large data, it simply won’t work. In these cases, what we want to do is submit our form with the body of the request (the query string is part of the head).

To submit as the body, all we need to do to our form is change the method it is using. Currently, it uses a GET request (as this is the default). We want to use a POST request instead. Adding the attribute method to the form with a value of POST does this:

function generateTitleForm() { return '<form action="/" method="POST">' + '<input type="text" name="title" value="' + title + '"/>' + '<input type="submit" value="Save Changes"/>' '</form>'; }

However, once we’ve made this change, our querystring approach no longer works (because our data is no longer in the query string). We’ll have to capture our POSTed data instead. This gets a little bit more involved, because the body can be any length - and it arrives piecemeal, through a series of data events, and finalized with an end event. We’ll have to reassemble it:

function parsePostData(req, res, callback) { if(request.method.toUpperCase() != ‘POST’) return callback(req, res); var body = ‘’; req.on(‘data’, function(data) { body += data }); req.on(‘end’, function() { var data = qs.parse(body); req.body = data; callback(req, res); }); }

Notice we use the callback pattern, as pulling data is an asynchronous process - if the request isn’t a POST method, we’ll hand control off immediately, otherwise we keep listening, and when we have the full body, we use the querystring library to parse the query string and attach the resulting object as req.body.

We need to enable this in our server:

var server = new http.Server( function(req, res) { var resource = req.url.split("?")[0]; req.query = querystring.parse(req.url.split("?")[1]); switch(resource) { case "/": if() parsePostData(req, res, serveGallery); break; case "/favicon.ico": req.writeHead(400); req.end(); break; case "/gallery.css": serveCSS(req, res); break; default: serveImage(req, res); break; } }).listen(80);

Then, in our createGallery() function, we’ll want to address the potential second source of data:

function serveGallery(req, res, query, data) { if(data && data.title) { title = data.title; config.title = data.title; saveConfig(); } ... }

Alternatively, we could merge the POST data with the query object - which many web frameworks do.

Protecting Against Data Overflow

There is also a security concern here - if a malicious user posts an extremely large body, we can chew up a lot of processing time working on it (or, if it was large enough or there were many such requests, we might exceed our computer’s available memory). One way to protect against that is to set a maximum size, test for that within our on data callback, and call req.connection.destroy() if the user exceeded the threshold. This would probably be a configuration setting we’d want to add to our config.json file as well, i.e.:

{ “title” : “Image Gallery”, “maxUploadSizeInMB”: 2, “author”: “Me” }

And in our parsePostData():

function parsePostData(req, res, query, callback) { if(request.method.toUpperCase() != ‘POST’) return callback(req, res); var body = ‘’; req.on(‘data’, function(data) { body += data; // Protect against huge file uploads if(body.length > config.maxUploadSizeInMB * Math.pow(10,6) ) request.connection.destroy(); }); request.on(‘end’, function() { var data = qs.parse(body); callback(req, res, query, data); }); }