Choosing the right worker queue service

Choosing the right worker queue service

Written by fdenkens Posted on November 18th 2015

The world changes quickly. Not only product software but also middleware applications are provided in the cloud or as a service. So we as operations people need to adapt.

Customers want to know what the difference is between the offered products. Trying to understand the pricing difference and make the right decision.

When giving advice you can’t check product features. But you need to check the different services they provide.

They differ between DIY, cloud and SaaS products. Vendors do their best to make it difficult for you. And it seems like you’re always comparing apples to oranges.

The world of worker queues is one of those mysteries. How can you compare?

Solutions

Everyone needs queuing to offload work from their frontend serving servers. Practically all web application has a queue feeding some backend process which can be processed asynchronously.

DIY

With DIY I mean install an existing queue solution on your own infrastructure. There is a comprehensive list at queues.io. DIY brings several layers of complexity you have to take into account. But in the meantime it provides you with the most flexibility.

Benefits

Every tool solves messaging in a different way and has a different set of features. You need to analyze all the different tools on the market to make sure you choose the right one.

Depending on what you expect of your tool there are different options you can choose. This is something we can help you with. We have studied and implemented several queuing solutions.

Pricing

Setup
Setting up a server is nothing more then and apt-get install away. The headache start when scaling it. It will take time and a lot of testing to do this right.

Some of the things you need to take into account:

  • Scale when needed to keep the costs low
  • Make sure it’s set up in a robust way that if 1 server is gone there is nothing lost and everything keeps working without losing messages

Operations
You have to do the operations:

  • Setup monitoring and alerting
  • Patching the servers for security updates
  • Wake up in the middle of the night if there is an issue

Our recommendation

Before you choose the DIY solution make sure you know all the pro’s and the cons. Make sure there is a solid reason why you would go this route.

SaaS

The complete opposite of DIY is just use a SaaS solution. This is the most easy way to go. Everyone from setup to operations is taken care of by the 3rd party. This also means less flexibility.

Benefits

You click signup/payment and start integrating their API. That’s how simple it is.

Besides all the features the main benefit is that there are no scaling worries or sleepless nights as the Saas provider takes care of everything. But you only get what they offer at the price they offer it.

Pricing

Let’s take 1 random example ironmq. SaaS companies are smart they don’t try to push you in a higher price bracket using consumption . They use features.

Look at the following as example:

If you need larger size messages but have less than 10M API calls. Pay more. More security? Pay more. Clustering and auto DNS failover? Pay more. The list goes on.

Because they know most of their customer will be companies with a smaller consumption footprint. The upsell happens by adding more features.

Second pricing issue is that you have to calculate the pricing of outgoing and ingoing data transfer at your current hosting provider. In most cases this is not that much but it’s an additional cost most people forget.

Our recommendation

You have to be careful to go this route. It can be expensive, and there are a few points you have to watch out for.

Security
Their not on your private subnet. You have to open you processing backend servers to the internet. You can off course limit this and secure it. But if you prefer to have everything on your own subnet this is not the way.

Network performance
Last but not least disadvantage is that the traffic is not local if you have high volumes on short time it all has to pass the internet increasing the latency.

Cloud service

The simplest service is Amazon SQS. Very limited in the features but cheap and easy to setup. If you’re already in the AWS ecosystem even better as data transfer stays on the same network.

Features

Amazon’s SQS has some quirks in their design but they’re very scalable and robust. There is a small added cost to catch the quirks. You need to think more about your application design.

With some simple application design you can use this robust solution and don’t have the headache of the DIY solutions.

Pricing

How much does it cost at Amazon? I’ll compare it to the production pricing of ironmq which is $ 499 and calculate the same volume. On AWS you’ll pay $ 220,35. Is it a fair comparison? No because I’m comparing apple’s and oranges here. But if you accept the way AWS has implemented queueíng it’s cheaper.

Our recommendation

If you can design your application to use AWS queuing and understand how to avoid the quirks. This solution is for you. Scaling is no longer an issue and you’ll only pay what you use. You don’t even have to know what you use and choose the right usage profile. It’s just pay as you go.

Conclusion

As you can see there advantages and disadvantages on all 3 solutions. As long as you’re well informed you can take the right decision.

We can help you make the right decision and set up the right solution for you.