Allen Eubank | Thoughts and Essays

Background Jobs Processing

There are plenty of packages that now support and enhance the ability to do background processing in Meteor applications. Background jobs are an essential part of a web framework. Many times I have had to set up background jobs to be run on the server, i.e. refresh a cache, process images, calculate stats, etc.

In Meteor, there are already a lot of options. Here I am going to demonstrate the most straight-forward solution to background processing.

Meteor.setInterval

There is a special method, Meteor.setInterval, which like the window.setInterval in the browser executes code repeatedly over a specified interval. Here is what my application initially looked like when I first attempted background job processing in Meteor.

// Runs every 10 seconds
Meteor.setInterval(function () {
  console.log('do your work, behind the scenes');
}, 10000);

Initial problems

Now, I wanted to set up my background jobs to be run on the server instead of the client,(I didn’t want to turn my client’s computer into a node in a giant server farm). The only problem with calling Meteor.setInterval on the server is that, because of the single threaded nature of the Meteor server, this essentially causes a block and the server is not able to handle any new requests while running running the Meteor.setInterval function.

Using npm package queue-async

In my searches, I found that there is a node package named, queue-async. Pairing this with the Meteor package, meteorhacks:npm, I was able to run the Meteor.setInterval function in the background, not causing the main Meteor server thread to block. Here is the relevant code to get that working.

// Run these two functions every five seconds
Meteor.setInterval(function () {
  var queue = Meteor.npmRequire('queue-async');
  queue()
    .defer(functionOneName)
    .defer(functionTwoName);
}, 5000);

Taking Background Job Scheduling Further

I found that this solution is great for functions that should be run without any knowledge of anything else and is simple background processing. The only catch when scheduling background jobs is,

Is it acceptable to have a type of background job overlap its execution with another of the same type?

In my case, I needed to be sure that only one type of this particular job was running at any given time. For this, I created a simple mutex that would lock when this job would start and would be released when that job was finished, pass or fail. Here is what that looked liked.

// Try to run function every 30 seconds, but only run one at a time
var isProcessNameRunning = false;

Meteor.setInterval(function () {

  // check the mutex, cancel this call if already locked
  if (isProcessNameRunning) {
    return;
  }
  
  // grab the mutex, locking this process
  isProcessNameRunning = true;

  // call the process
  var queue = Meteor.npmRequire('queue-async');
  queue().
    defer(processName)
    .await(function (error) {
      // process returned, free the mutex
      isProcessNameRunning = false;
    });
}, 1000 * 30);

Fin

Thanks for reading, hit me up on twitter for any questions/comments/ideas.

@alleneubank