Handling a Compressed Response in Node.js

So I ran across this problem when trying to consume a RESTful service. I couldn’t figure out why I was seeing nice response bodies in Advanced REST client but in the Node debugger, a bunch of noise. Eventually I saw this in the response header:

'content-encoding': 'gzip'

That’s when it dawned on me what was happening. This response header was telling me that the server was sending back compressed data. So I spent some time chasing down how to gunzip in Node. Interestingly, my requests never asked for compressed http. A brief overview of http compression can be found here.

The example below removes all dependencies on third party libraries and focuses on solution free from distraction. It uses Node’s Http and Zlib objects which are part of Node (v.10.22 at the time of writing this). If you are using Express or some other modules to handle your Http requests, you’ll need to get your data into Zlib in a slightly different way but the approach should be similar.

var zlib = require('zlib');
var http = require(‘http’);
var req = http.request(options, function (res) {
          res.on('data', function (data) {
              debugger;

              // for some reason response from this route is gzipped...
              // see the header 'content-encoding': 'gzip'
              // so unzip it.

              zlib.unzip(data, function(err, gunzipped){
                  if(err){
                      // log the error with your logging utility
                      // respond with an appropriate error.
                  }

                  var message = JSON.parse(gunzipped);
                  // do something here with your gunzipped message body
             // send a response
             });
          });
      });
      req.write(JSON.stringify({<< Some response body here >>}));
      req.end();
      req.on('error', function (e) {
          debugger;
          // log the error
          // send an appropriate response.
      });
  };

The trick lies in the zlib.unzip method. It expects the data from the http.request’s onData event to be passed in. Once gunzipped, we can do what we want with it. In my case I am expecting JSON bodies so I parse it into an object I can use.

In the rest of the example, the req.write, req.end, req.on(‘error’) are standard approaches to using the Http object to make a request to some end point.

It might be a good idea to check the 'content-encoding' header before gunzipping, even when you expect it. The service provider could change the the header underneath you and cause a problem. The service provider shouldn’t make such a change but that discussion is another matter entirely.

Multi-Part File Upload with Node JS and Express

Several examples exist on the internet that detail how one can use Node.js to handle a multi-part form upload. Well, the scenario I ran into was that I needed to expose the upload functionality in my RESTful service, but then needed to pass that upload along to another service. So I had to construct the request object via code. Typical request objects are not so hard to construct in Node.js but the multi-part form upload was tripping me up. Here’s what I eventually came up with.

This example uses Express. Express is a minimal web application framework. You can find it here: http://expressjs.com/. Express exposes the collection of files via the req.files object. Note that express is going to use the name assigned by request creator - so, part of the contract may require the name be a specific value, otherwise there is no way to glean what the user may have named the file - at least not that I have found so far..

We have to tell Express to use the body parser. Do some googling on the bodyParser, there are some properties for specifying the temp directory for the upload and other things as well. Adding this line will cause Express to grab and expose the files as part of the request:

app.use(express.bodyParser());

And here is the POST handler:

app.post('/me/avatar', function(req, res){
   logger.debug('handling POST /me/avatar');
   logger.debug('files :' + JSON.stringify(req.files));
     
  
   // ToDo: this is a problem. It looks like we have to force people to use 'upload' as the form name for the file.
   // Cannot find away around this after hours of searching. Probably something simple,
   // just can’t find it (node.js noob).
   if(!req.files.upload){
      res.statusCode = 400;
      res.send("File not found in request.");
      return;
   }

   var subServiceUrl = “http://some_service_i_need_to_upload_to/my/file”;
       
   // read the file from disk. It has already been uploaded to a temp directory
   var filePath = req.files.upload.path;
   logger.debug('reading file: ' + filePath);
   fs.readFile(filePath, function (err, fileData) {
      if (err){
         logger.debug("error loading file: " + filePath);
         logger.debug(err);
         res.statusCode = 500;
         res.send('');
         return;
      }
   
   
      // the options setup is the tricky part for the multi-part upload
      var options = {
        uri: subServiceUrl ,
        method: 'POST',
        headers: {
          'content-type': 'multipart/form-data'
        },
        multipart: [{
           'Content-Disposition': 'form-data; name="upload"; filename="' + req.files.upload.name + '"',
           'Content-Type': req.files.upload.type,
           body: fileData
        }]
      };
   
      request(options, function(error, response, body){
         logger.debug('statusCode : ' + response.statusCode);
     
         if((response.statusCode != 201) || (response.statusCode != 200)){
            sendError(body, res, response);
         }
         else{
           res.statusCode = response.statusCode;
           res.send(response.statusCode, body);
         }
      });
   });
});