[Farmware] Loop-Plants-With-Filters

Ok, thanks very much for detailed explanation.
So the move command can run with a variable looping on a group of plants, and that does the job (i successfully tested it)

Next step is to filter and update the plant status in state “planted” (or whatever) when running such loop, to be able to recover from a glitch. I hope your next release will be able to do that ?

3 Likes

Next step is to filter and update the plant status in state “planted” (or whatever) when running such loop, to be able to recover from a glitch. I hope your next release will be able to do that ?

Yes, this is what we are working on this month. There are still some issues for us to fix on our end with the UI, but we have tested it on internal builds and have it mostly working. The release will be announced on our newsletter very soon.

Thanks @RickCarlino for that wonderful step-by-step instruction. It probably was a bit confusing for most people, especially for those who did not see the software update announcements.
What do you think about adding this at some time in the “Tours” section of the WebApps help page?

1 Like

That’s a good idea @Ascend. We’re also going to update our documentation, since this is the second time the question has been asked this week.

Thanks again for the explanation, a doc update would be nice.

The sequence on a plant group works, but it still does not update the plan status from “planned” to “planted”, and it cannot recover from a glitch.

But so much technology from a California company working with the NASA to change the world, 9 versions of software, and it is still not possible to update the plant status to properly manage 40 seeds of radish, that’s a bit ridiculous !

How can i explain to my 12 years old son that the farmbot can do that, just a loop with status update on each step ? So please quickly fix that or share the next release code on github, we are contained because of the COVID19 and this is time to make this work properly, time for misleading advertising is finished.

1 Like

@vroyer it sure is time for better software !

I think @RickCarlino would agree that Farmwares should not be cut off after just 60s !
Now that the old Farmware v2 execution bugs have gone, a 5 minute “safety stop” limit for a running Farmware should be totally fine !

What’s a typical end-to-end run-time for your Farmware ?

[edit 0] I personally don’t like Farmwares . . there’s very little you can accomplish while the Internet connection has dropped !!

[edit 1] Why not make this “Farmware run-time timer” configurable !!? somewhere.

@jsimmonds @roryaronson The Loop-Plants-With-Filters farmware was good to build complex sequence using a filtered list of plants, and update plant status while sequence is progressing. With the farm designer v9.2, you can’t even manually update the plant status !

Give up this opensource farmware support while the farmbot software cannot do the same or the equivalent (manage sequence with loop, status filter and status update), it’s just an incomprehensible strategy for an opensource community…

1 Like

@vroyer, as you’ve noticed, some farmwares from the past no longer work with the latest versions of FarmBot OS and require updating to work. This is because we had to make some breaking changes to the way farmware works within FarmBot OS, which is why we incremented from v7 to v8 (marking a major release).

While nobody ever wants to intentionally break integrations, plugins, compatibility, etc, sometimes it is the right decision to make to push the greater platform forward. When we were making this decision, we first looked at the number of farmwares actively installed on our customers’ FarmBots. You might be surprised that the number of unique farmwares that were being used back when v8 came out was less than 10, and the number of FarmBots with those farmwares installed was less than 20. While we immensely value the importance and contribution of our “power users”, we also must recognize that they make up a very small fraction of our customer base. So we must balance the effort expended supporting this small subset of users with the effort spent supporting the vast majority of our users who are not software developers and/or have no interest in finding and using publicly available farmware.

During the v8 build phase, we emailed farmware developers we knew about and discussed plans for how to move forward together. As with most decisions in software development, we had to make tradeoffs. In order to move the core FarmBot feature set forward (adding support for sequence variables via the new “celery script runtime”) farmware would have to be updated.

Unless FarmBot Inc has authored the farmware, we don’t have any control over a 3rd party author updating a farmware other than politely asking. If the author is unable to make an update, and if the farmware is open-source, someone else may feel inclined to pick up where things left off, though nobody is making any guarantees that that will happen.

I hope this sheds some light on the current situation and allows you to understand that we didn’t give up some farmware support for any nefarious reasons, or for no reason at all. We have arrived at where we are today after much deliberation and a lot of calculated decision making.

In terms of FarmBot’s open-source strategy, we are doing everything in our power to support and empower people to make FarmBot better, and are always open to feedback (and PRs!). We not only have licensed all of our technology with permissive open-source licenses, but we have gone above and beyond in providing high quality documentation, CAD models, schematics, a public forum, an open API, hackable hardware, a parts shop, an app experience designed with customizability at its core, and a 3rd party plugin system (farmware). We have done all of this because FarmBot is a very ambitious project that has many moving parts, and our official team is very small. There is no way our team, or any team, could ever realize the full potential of FarmBot. Only an open ecosystem with a strong community that is empowered to make their ideas reality will achieve that.

Okay, so what about these features you want? We want them too! And we’re working on them - every day - and have been building the foundation to get here for years now. We’re also balancing the work on new features with improvements to stability and performance, adding support for our latest electronics, and fielding the myriad of other requests we receive both here on the forum and through our official support email. Check out our GitHub repositories and public project board to see the daily and weekly progress, and feel free to hop in yourself. PRs are welcome, and we’re happy to help anyone who is serious to get started with development.

We also had a meeting yesterday to figure out how to expedite the release of groups based on filters (all weeds, all broccoli plants, all plants that are ‘planned’, etc), and are working on getting out an initial implementation as soon as possible (we’re shooting for next week). After that, as you will see in our project board’s TODO column, next up is improving the MARK AS command to support variables, and a write-in custom option. These features will allow for a very customizable “programming” experience that will offer similar functionality to some of the most popular farmwares.

Additionally, I spoke with Rick this afternoon and we will be lifting the 60s limit on farmware with the next OS release. To be clear, we have not and will not be testing long-running farmware and your mileage may vary. You may run into unexpected issues and your FarmBot may become unstable. We’re doing this because of the feedback in this forum post and to support you all in tinkering. However, we simply don’t have the resources at the moment to fix issues that may arise after 60s of execution, but again, open-source contributions are always welcome.

3 Likes

As Non-One has yet jumped in and created a “FarmBot NG” post to discuss how Customers and Happy Hackers want to see the FBOS Road-Map (!) I’m creating a new topic for FBOS and WebApp futures from the consumer side :slight_smile:

FarmBot software post-v9 wish-list

Well-written Rory and thank you for the explanation! Forgive me if this has been discussed, I have been out of the loop for awhile (ha!): could the script timeout be settable by the user per script (with a default of 60 secs)? I realize this could likely involve a deeper architecture change. From a regular use perspective, they’ll never know. From a power user perspective, it’ll be useful for chaining Farmware scripts. To your point, those choosing to play in the realms that are longer than 60 second are swimming in the open ocean with no expectation that a lifeguard (or coast guard) will come to help.

@jsimmonds I’ll add my suggestion to your roadmap topic.

1 Like

@roryaronson Thanks for your explanation.

-The issue is not that you introduced breaking changes in the farmware, this is real life of evolving software.
-The issue is not your farmeware support, the community (or even me) can probably fix breaking changes.

In previous releases, the opensource Loop-Plant-With-Filters farmeware was the only workaround to to build generic sequence to grow plants (plant seeds, water, or whatever) with status management, but due to the 30s or even 60s timeout, it’s practically unusable.

The Loop-Plant-With-Filters farmware already does through your REST API what your Farmbot-Web-App should be able to do. So i can’t figure out why it is so complicated to move or rewrite that piece of code in your farmbot-web-app or even make the farmware timeout customisable as suggested by @fuzzynickel and ship the Loop-Plant-With-Filters in your release until your software can do it better ? An imperfect solution is always better than no solution.

Thanks again for your long feedback, but you did not give me any solution, no release date, no workaround … just a TODO project card for a partial feature not planned.

FYI : In current FBOS release v9.2.2 @RickCarlino has extended Farmware runtime max to 20 minutes

1 Like

@vroyer today we released a new version of FarmBot OS and the web app that includes our first implementation of filter-based groups. Check out the details here: April 13, 2020 Software Update

And yes, as @jsimmonds pointed out we lifted the farmware runtime limit to 20 minutes in case you still want to use long-running farmware in lieu of or in addition to the 1st party feature set.

I disagree on this point. We have made imperfect solutions available before. Some have been worthy interims, while others have had bad consequences. In some cases a half-ready solution has caused our customers confusion because the UX wasn’t straightforward, or frustration because there were bugs. In other cases a rushed temporary solution can end up delaying the final solution because we need to worry about data migrations later on, corrupted data, regressions, the negative feedback of sunsetting something people have gotten used to… And in almost all cases, any work that is not aligned with our longer term roadmap is likely going to get scrapped when the real solution is ready. So all the time spent developing, testing, announcing, documenting, and supporting that work was for a short-lived feature. Usually effort is better spent working on stuff that will have a long life and builds the foundation for future features.

Moving or rewriting Loop-Plant-With-Filters into the main codebases could have been done, but it would not have been the right long-term solution for the core platform. To be completely honest, Rick had suggested that idea a few months ago to me and I turned it down, wanting us to focus on the real deal. I felt that implementing the initial version of manually created groups (released in October) would be a decent enough interim, and again, aligned with our longer term roadmap. I stand by that decision.

As mentioned previously, our next major feature will be improving the MARK AS command to work with variables and offer custom write-in options. While I haven’t used Loop-Plant-With-Filters since shortly after it was first released, I think the core feature set will more or less match that farmware once the Mark As command is upgraded.

@roryaronson Thanks for this new release. I have successfully tested the new dynamic group feature, although i now get continuous errors and disconnection (see screenshot). I hope the MARK-AS command will be able to update the plant status and update the associated date to get the history of plantation ?

Moreover, it’s seems you don’t have a complete vision in automating cultivation. For example, if someone wants to plant 1 lettuce every 2 days to get one lettuce every 2 days one month later. How can i schedule such job with your dynamic groups ? We need an ordered group to pick the next N plants in a group, and execute every 2 days the sequence “plant 1 seed from group1 with status planned”…

Same remarks about weed detection, the farmbot should be able to scan all the grid, then detect weeds on pictures and finally run a periodic sequence to destroy weeds, but how can you loop to cover 100 of the grid ? Write a long sequence with all the camera positions ? Then should i run the weed detector farmware at each step in the sequence or later on the picture set ? I think weed detection should be done in the cloud with various testable detection algorithms, not on the small farmbot harware ?

The Loop-Plants-With-Filters was release 2 years ago, and you still seems very farm to automate cultivation. Hopefully, you have increased the farmware timeout and we can probably still rely on it, but if you want some contributions, you should share your technical vision for automation instead of dropping a new limited feature every 6 months.

@vroyer I understand your last response wasn’t addressed to me but there is a feature request thread where Farmbot are evaluating use cases and solutions from customers regarding the future of Farmware and the like.

Please feel free to read and contribute to this feature request FarmBot OS and WebApp post-v9.

What was your test ?
Do you believe this test has caused the low-level comms problems ?

@vroyer

continuous errors and disconnection (see screenshot).

Your account has 33% WiFi strength right now. We’ve never seen a FarmBot successfully operate with that low of signal strength. My guess is that the WiFi connection is not responsive enough to keep an AMQP connection alive. How long has the unit been operating in its current location? Has it operated correctly until recently? These socket errors are less likely to occur with a signal strength above 55%. Most every customer that has noted socket disconnects has been able to fix the issue through the use of a WiFi extender or by relocating the WiFi access point if possible.

hope the MARK-AS command will be able to update the plant status

The next big feature that we will release and the one which I am actively working on this month is the ability to pass a variable into a MARK AS step and update arbitrary properties on it (such plant_status, meta, etc…)

weed detection should be done in the cloud with various testable detection algorithms, not on the small FarmBot hardware

This is our long term goal and we have actually run some early internal testing on exactly this use case (via Google’s AutoML). As you’ve mentioned several times already, the development pace is slower than we would like and this is due to the fact that FarmBot is a small company that only employs 2 full time software developers. Our long term vision of the product will take time to accomplish given this reality.

Hi Rick,

Yes, wifi strength is sometime 30%, sometime 100%, it seems to work better today.

I few question about the network:
1-Why Voltage is yellow, is it an issue ?
2-Ping, is it the IP protocol ICMP, or TCP or Application ping ? So latency include the internet latency from France to the datacenter where the webapp+mqtt are deployed ? Internet is currently heavy loaded du to the COVID19 containment, and without flow control from TCP, such ping is not reliable.
3-The farmbot should be able to work offline, or with network having high latency because farming is basically a low-tech. Moreover, the farmbot should automate 3 main recurring tasks : plant, water and weed. So, sequences should be uploaded into the farmbot, run unless there is an issue, and data should be uploaded in the cloud asynchronously (plant status update, pictures, etc…). The farm designer should be able to run offline, and then you deploy your “farm-plan” to run once or continuously (like a control loop).

About plant status, you should keep the update date with each status in a status history. This will allow to do statistic on cultivation, and sharing informations between farmbot users would help to identify better practices in your farmbot community (You should create the social network of farming, the facebook of personal local and bio farming).

For plant detection, just gather images properly in the cloud, and create a ML challenge on https://www.kaggle.com/ to detect weed. Many specialists can help you to efficiently recognize plants. Moreover, you should not only detect weed, but also identify germinated seed of what you have planted. So, the algorithm should automatically detect weed and planted vegetables. It is a supervised image classification algorithm, and again, you community can help in classifying images…

I also founded a small company delivering opensource software (www.strapdata.com), and i’m also working hard like you guys, but if you really want to change our crazy world, this is the minimum !

1 Like

@vroyer I’ve provided a response below. I am going to lock this thread now, as we are no longer discussing loop-plant-with-filter. You are welcome to create new topics if you wish to continue the discussion.

Why Voltage is yellow, is it an issue ?

This was a bigger problem for DIY setups and kits <= v1.2. It is rarely a problem in 2020.

Ping, is it the IP protocol ICMP, or TCP or Application ping ?

This is an application ping via a round trip MQTT/AMQP echo message. We send a timestamp to the device over MQTT and then perform a diff on the timestamp when the device echoes it back to the client. It is preferable to traditional pings because it also checks if the AMQP/MQTT ports are blocked (common in schools).

The farmbot should be able to work offline

If your device has a realtime clock (>= v1.5) it should be able to perform all basic sequences / farm events without an internet connection. The exception is “write” operations such as MARK AS or take-photo.

While the device is offline, you would obviously not be able to send commands, change settings or update existing resources but the FarmEvents should continue to operate. If you have an RTC and still experience issues, please let me know (ideally, after the device comes back online and I can remotely view error logs).

Devices that do not have an RTC will need to stay connected to receive a clock signal via NTP, otherwise clock skew will eventually stop events from running reliably.

data should be uploaded in the cloud asynchronously (plant status update, pictures, etc…)

This is more possible now that devices have real time clocks, but there are still some challenges. For things like photo uploads, it is a simple fix: just create a queue and drain the queue when the network is available.

For other operations, such as MARK AS, things become more complicated because you have multiple partitioned data writers and taking a “last write wins” approach could introduce some very confusing results to end users. If we were marketing this product to technical users only, it might be easy to fix, but FarmBot is a consumer product and that means we can’t expose complicated concepts like eventual consistency, delay tolerant networking and merge conflicts to end users.

For example: a user edits their garden while their offline device is running a “MARK AS” operation. When the device comes back online days later, a decision must be made as to which information is “most authoritative”. We thought about adding a diff viewer, but are opposed to the idea because it takes us away from the long term vision of fully autonomous operation.

The options we’ve considered so far are:

  • Last write wins :-1: - Confusing experience for non-technical users.
  • Diff viewer :-1: - Confusing, not aligned with long term vision of fully-autonomous operation.
  • Row locking :question: - Might work, but could lead to surprising errors when the device is offline for long periods.
  • Something else :question: If you are aware of a partition-tolerant data store with a history of embedded systems (not server) use, I’d be interested to hear about it. We’ve tried running traditional databases on the device with poor outcomes. The current DB is SQLite. We’ve had our eyes on CouchDB but have not investigated extensively yet.

you should keep the update date with each status in a status history.

This is on our road map. Like everything else, it will take time given the realities of our company size.

2 Likes