Revise the task queue mechanism

Currently, it seems impossible to reliably schedule multiple Farm Events at the same time and expect all of them to execute. This seems counter to the vision of FarmBot, which is primarily making growing your own food using automation accessible to everyone.

Regimens are designed to specify how the FarmBot cares for a particular plant type. Regimens don’t care if it’s winter or summer, they care about what “Day 1” and “Day 12” etc. look like for a given plant type.

Imagine you are preparing your FarmBot to handle many different types of plants, some of which you’ll plant during winter, some you’ll plant in spring, and some in summer. Some plants you might grow all year 'round, or repeatedly every season.

You’d make a Regimen for that plant type and specify day-to-day how to grow that plant from a seed to harvest. And you’d do this for (say for the sake of this example) 25 plant types, 25 Regimens.

In the current Farm Event scheduling mechanism, you’d have to manually ensure that the start times of your sequences inside each Regimen do not conflict with the start times of sequences inside other Regimens. That is, in my opinion, an unreasonable expectation. I would like to propose a conceptual solution that I believe would be possible with FarmBot, although potentially requiring some significant refactoring of the existing scheduling logic.

The Queue

The Queue is a list of tasks that is kept separately of all other current scheduling mechanisms. It becomes the primary source for FarmBot to periodically check if it needs to perform a task. Tasks are added by Farm Events and by the user requesting to run sequences ad hoc. The Queue does NOT have a concept of time, it simply exists to hold tasks and its goal is to be empty.

Benefits

  • Currently, when FarmBot is executing a task, it’s not possible to instruct it to run a sequence without interrupting the current task. With the Queue, clicking Run on a sequence would essentially queue the sequence as the immediate next task, regardless of how many tasks are currently already in the Queue. This allows you to inject ad-hoc tasks into a big list of tasks without having to cancel and restart all those other tasks.
  • Currently, scheduling Regimens with Farm Events that have sequences begin at the same start time means that only the first couple of operations will succeed. All other remaining tasks from that start time will simply disappear without being executed. With a Queue, all Farm Events will simply add all their tasks to the Queue when the Farm Event fires. The Queue then takes care of the sequential processing of these tasks, even if it takes 4 hours to reach the last added task.

Some constraints

  • The Queue must work on a First In First Out principle. The only way to “jump the queue” is when a user clicks Run on a sequence. This will insert the sequence directly after the currently executing sequence, if there is one.
  • Farm Events firing at the same time must add their list of tasks contiguously, meaning that tasks from one Farm Event may not blend into the sequential list of tasks from another Farm Event; each Farm Event is contiguous. The reason for this is because some sequences rely on other sequences to have executed, like mounting a watering nozzle before watering.
  • FarmBot is a machine and does not care how long it takes to perform its tasks. But to manage user expectations, a periodic health check must be performed. This health check is simply verifying that the Queue was empty at least once in the last 24 hours. If it was NOT empty in the last 24 hours, it means FarmBot has too many tasks scheduled that cannot be completed in a full day. The health check could be performed as often as every minute, at the same time it checks if there are tasks in the Queue that should be picked up.

These are all good points, @mdingena.

Quick disclaimer: Before I address this proposal, it’s important to remind anyone reading this thread that there are only two software developers on staff at FarmBot, and we’re still independently owned and operated (compare this to the company size of any other IoT product in your home). Sometimes when feature development moves slowly, people may conclude that we’re refusing to develop certain features. Oftentimes, it’s simply a matter of resource allocation. A good example is the criteria-based groups- it was released about a year ago (2020), but it has been on the roadmap since 2014. FarmBot in 2021 is still very much an underdog in the IoT space, and this isn’t necessarily a bad thing. Being small has helped us keep our interests aligned with users. Often this comes at the cost of feature development that is slower than anyone would hope for. I thank everyone in the community for their patience as we improve the product.

OK, now for a real response…

Concerning your “queues” idea - we are heavily considering this option. We haven’t written too much on this publicly, but we’ve spent hours/days discussing it, and it is probably the solution we are going to have moving forward. It won’t be exactly like the solution you are proposing, but it will be conceptually similar.

We have plans to overhaul the farm event system, including a “task manager” GUI that offers facilities to manage past and upcoming events. We’re also considering moving event scheduling into the API, allowing for more consistent scheduling and the ability to have “watchdog” alerts when events do not execute as expected. Example: Your bot can’t report a failure if it crashes or experiences SD card corruption currently, but this is very simple to do if the API is partially involved with task supervision.

With regards to regimens, you are absolutely right that they need some updates. There have been two factors that have kept us from allocating many resources to a Regimen system update. When we analyze real-world data of long-running devices, we find that most users are not using the Regimen system. Regimens are very popular with power users, but statistically, the average user prefers to use FarmEvents. Additionally, If you couple farm events with criteria-based point groups, it is possible to accomplish many of the same goals as regimens, at least in the interim, until more time is devoted to a Regimen overhaul.

I hope this answers your questions. Thanks for sharing your ideas with us!

1 Like

Thanks for your detailed response.

Have you considered users aren’t using Regimens because without a rework, they offer little value? As a power user, I find they offer me little because they do not fully work as intended.

My prime example is having a single Radish regimen that 10 groups of 3 radish plants use. With a Farm Event, I stagger the start date for each group but they all use the same regimen. The problem that arises is that they all use the same time-of-day to perform their tasks.

This fact, in combination with the scheduling mechanism, makes it so Regimens are not useful to me currently. So I will likely stop using regimens which only strengthens your belief they are not useful.

But you are wrong (respectfully). I think Regimens and Farm Events are probably the two killer features of FarmBot and users must be educated on their use. Give users some radish seeds with their kits and instruct them to grow staggered groups of radishes. They’ll quickly understand why regimens are so great.

But, one more thing that needs to be fixed is being able to specify which subsequences should loop and which shouldn’t (see Nested sequences and Plant Groups), because currently my bot is mounting and unmounting the watering nozzle for every single radish and there is no way around that at the moment!

1 Like

@mdingena I do agree that we are in a chicken-and-egg scenario with Regimens. With regards to the thread you’ve linked to, I’ve provided a response to your original post in the link, and hopefully, it gives you a better idea of how things work.

1 Like

To confirm, I have stopped using Regimens because of :point_up_2: this problem and switched to manually scheduling Farm Events with a guestimated time gap of 5 minutes between each event.

image

I hate it… I can’t even see for which plants the sequences are (Regimens showed you the Sequence name and below it, it showed which Regimen put that Sequence there).

Please considering giving Regimens the love they deserve. I know your resources are stretched really thin but Regimens are fundamental to FarmBot operation.