Queued items service limit

Automation for Jira comes with a number of service limits, to protect the performance of your Jira instance. One limit specifically guards how many items a single rule can add to your automation queue for processing. This KB explains the reasons behind this limit and provides a number of ways to update your rules to reduce the likelihood of hitting this limit.

Background

Automation for Jira uses a rule processing queue behind the scenes to manage the execution of automation rules in your instance. Jira only has limited capacity available to serve requests from users in the browser, as well as executing 'background' services such as automation rules. To ensure we don't overload Jira and make it unresponsive for real users, we queue automation rule executions and limit how many items we process off this queue in parallel. By default in Cloud we only process 8 items in parallel per Jira host, and 6 concurrent threads in server (per node for Data Center configurations).

Our automation rule builder is very powerful and allows users to configure rules that do just about anything. This can result in rules that are very expensive to run. Lets consider a particular scenario with these example issues:

  • ABC-120 - Parent development issue with issue type 'Task'
    • ABC-121 - Subtask 1
    • ABC-122 - Subtask 2
    • ABC-123 - Subtask 3
    • ABC-124 - Subtask 4
    • ABC-125 - Subtask 5
  • ABC-130 - Another Parent development issue with issue type 'Task'
    • ABC-131 - Subtask 1
    • ABC-132 - Subtask 2
    • ABC-133 - Subtask 3
    • ABC-134 - Subtask 4
    • ABC-134 - Subtask 5

Now consider we have the following rule:

Sample rule

This simple rule, will actually result in 13 individual items queued in the automation queue. Lets break this down:

  • 1 item queued for the initial scheduled trigger execution to run the search
  • 2 items queued for the matching parent issues ABC-120 & ABC-130
  • Then for each of those items, the related issue branches will queue 5 items each

So that's 1 + 2 + 5 + 5 = 13 items all up.

You can see that these types of rules can quickly add up to a lot of queued items if for example the initial scheduled trigger matches a lot of issues, or if the related issue branches match lots of issues (related issue branches can also search for issues by JQL!).

Why are lots of items in the queue bad?

Our infrastructure can handle lots of items in the queue fine. However single rules that insert lots of items are bad for perceived performance of Automation for Jira. We've seen cases in our Cloud infrastructure where a single rule can insert over 100,000 items in the automation queue for that single Cloud customer. That means all 8 processors for that customer will be busy for hours processing this queue. During that time, other rules will be delayed.

We already prioritise certain rule executions to get around this to some extent (e.g. Manual rule triggers will be executed first), but this approach can only go so far.

Throttle and disable

As a result when we detect that a rule has added an excessive number of items to the queue at the end of its execution, we log this in the audit log and disable the rule to prevent future executions:

Queued limit exceeded

What types of rules can cause this?

Rules that exhibit this problem generally fall into a single category - a rule with lots of related issue branches, where each related issue branch matches a significant number of issues.

So for example a rule that could get you into trouble is:

  • Scheduled trigger that matches 100 issues
    • Related issue branch that matches 50 issues
    • Another related issue branch that matches 80 issues

This would roughly result in 13,000 items (100 * 50 + 100 * 80).

Another type of rule that might cause this is one that has a lot of related issue branches (to simulate if conditions):

  • Trigger: Issue updated
  • Related issue branch with JQL: type = Bug that matches 1000 issues
  • Related issue branch with JQL: type = Task that matches 500 issues
  • Related issue branch with JQL: type = Feature that matches 2000 issues
  • ...

This would result in 3,500 items or more depending on the number of branches (1000 + 500 + 2000).

Prevention

Finally we've arrived at the spicy part of this KB! Luckily most rules don't have to execute for quite so many issues on every execution. There's a number of options that can help reduce this problem:

  • Ensure you use JQL that limits the execution to the smallest possible set of issues. This can be achieved in a number of ways:
    • Ensure your JQL search is specific as possible. E.g. don't just search for issues that match type = Task if you only care about issues currently 'In Progress'. type = Task and status = "In Progress" would be a better search
    • Only include issues that have changed since the last time the rule executed by checking the Only include issues that have changed since the last time this rule executed checkbox. For many rules, it's perfectly fine to operate only on this small sub-set
  • Don't create related issue branches for 'conditional checks'. So the rule above with many related issue branches checking for type = Bug, type = Task, etc could be written more efficiently like this:
    • Trigger: Issue updated
    • Related issue branch with more specific JQL to match the issues you're after (probably related to the trigger issue somehow)
      • On this branch use the if/else block to match based on the issue type

In general the goal is to reduce the overall number of issues a rule will fetch either via the trigger or related issue branches.

If you're unsure about how to do this for your rule, please contact us in support!