MQTT live data (channel)

Feature Requests
1

Hello,

I’ve been developing the farmbot-py client that connects through MQTT. It works pretty well, but the one thing that is holding back a solid implementation is a good way to get at “live” data, and specifically the status of the mounted tool.

Currently there are two ways to get status information through MQTT: through the status tree in the status channel, and through the log messages in the logs channel. The problem is that the status channel is mostly sending configuration information - which hardly changes and is very uninteresting to send over and over - and that the logs channel is for logging, not an official interface. Therefore I propose the following:

  1. Make a distinction between configuration data and “live” data, and send each through their own channel.

  2. Define “live” data to send, which consists of:

    • x, y, z location,
    • a way to determine if x,y,z is unknown, mostly due to a reboot, and a homing is needed,
    • status of sensors,
    • status of readable pins,
    • overall readiness status (live, e-stop, etc)
    • ID of mounted tool or no tool mounted

  3. Define configuration data to send, which is most of the current data in the current status tree

  4. Split off jittery data, notably the wifi strength and raw encoder values, since these two make diffing two statuses guaranteed to produce a difference.

  5. Create a new “/live” (or whatever name is deemed appropriate) channel to send the dataset of 3) through.

  6. Decide on a timing for the live data to send. My preference is using a correlation ID that corresponds to a “please send me a new set of live data” request in the from-clients channel. (I know that that correlation ID would already be used in the rpc_ok message, but this is another channel so will correlate nicely)

  7. Gradually deprecate sending “live” data through other channels

2

@sbeaumont We definitely agree with you on this one. The current /status channel is extremely inefficient and is the result of some legacy baggage from old versions / infrastructure. We are actively working on a solution to the problem this week.

I’ve noted some of your concerns below:

Status channel bloat: We’re adding a new bot/device_123/status_v8/# channel this week that will use one MQTT channel per status tree entry, rather than sending a gigantic JSON state tree every time.

For example, if you are only interested in the Bot’s X/Y/Z position, under the new system, you would be able to subscribe to the MQTT channel bot/device_123/status_v8/location_data/position/#.

Your MQTT client would then receive a float value for X/Y/Z over the following MQTT channels (and nothing else):


bot/device_123/status_v8/location_data/position/x

bot/device_123/status_v8/location_data/position/y

bot/device_123/status_v8/location_data/position/z

This allows us to use MQTT as the protocol intends while also making existing applications more performant. It will solve the “jitter” you experienced and will also solve the matter of having excessive items on the status tree- you would subscribe only to the status entries that matter to your application.

Conversely, if you do wish to still receive all status updates, you could subscribe to the bot/device_123/status_v8/# channel.

After deploying the new /status_v8, we will gradually deprecate the /status channel.

Since we are actively building the feature this week, I would highly appreciate feedback from third party developers. I am eager to hear your thoughts on this feature.

Tracking mounted tools: We added a mounted_tool_id to the device resource on the API (example API request here). A new CeleryScript node will eventually be added to the sequence editor/RPCs called “mark tool as mounted”. This feature is finished but still needs QA (and is not available on production yet). Under this scheme, you would mark a tool as mounted after a particular RPC completes. This information is then available to other Farmwares via the device.mounted_tool_id property.

image
image.png766×148 6.53 KB

You could track mounted_tool_id updates by subscribing to the device resource on the auto sync channel or downloading the device resource from the REST API. We have also considered duplicating this field onto the device status channel to support users with slow or unreliable network connections.

Differentiate configuration from “live” data: Configuration parameters are now stored on the REST API, (as fbos_config, shown here) since configuration is a data storage concern. The remaining items will eventually be removed from FBOS, but there are still some use cases where the bot must keep track of configuration locally. This is the result of edge cases while factory reseting or for farmwares that have a hard requirement for offline support. For your use case, it sounds like the new /status_v8 channel will address your concerns. Please let me know if there are still problems that /status_v8 would not be able to address.

Would the solutions noted above help? Please let me know. If you would like to have a more in-depth conversation, I would be happy to schedule a Skype call over private message.

Also, I am not sure I understand item #6 in your proposal- are you sugesting that we add “return types” to RPCs rather than using out-of-band responses? This is something that @connor and I have both considered, but we have not started work on it or finalized anything yet. Feedback appreciated.

1 Like
3

Hi Rick,

Yes, the status_v8 sounds like a huge improvement! I’ll be happy to try it / test it. Are there any special things I need to do, or is it automatically updated and deployed?

In your response I haven’t seen an explicit answer to how “read pin” information comes back, and how I indentify that this was the status after the “read_pin” command.

With respect to mounted_tool_id:

Does this mean that there is no hardware way to identify the mounted tool? The screws in each tool suggested to me that they’re some sort of binary code that identifies it, but apparently that’s not what they’re there for.

Although it’s nice to have a place to at least persist it, it can’t be a final solution. When the farmbot reboots, which is needed regularly in our case, you can easily reset the knowledge of the location by homing. But for the mounted tool no such solution exists that I know of. For instance, if the bot crashed while seeding, I’ll have to know the seeder was already mounted, otherwise I have no way to know what the correct movements are wrt the tool bay.

With respect to your question re my #6 proposal:

For me it’s not about a specific solution, it’s about being able to get at live information from the farmbot through MQTT without needing to go through the web api: after all, the web api had to get it through MQTT as well, so it would be an unnecessary detour.

My use case is that I need to be able to identify when the farmbot is in a state that it can process a next command. A return message with a correlation ID works well, but in the case of an out of band response, I’d need a solid way to monitor the changing live data so I can confidently determine the bot is ready for the next command. This is the main reason the jittering data was an issue, since I wanted to diff the status tree versions to see if there was a state change, and the jittering constantly triggered it. To summarize, somehow I need to be able to see that “here is the state of the location / pin X / other live thing that resulted after you did the move / read pin / etc command”.

A Skype call would be fun: my and my colleagues working on the bot could show you our implementation and would probably enjoy a good in-depth discussion.

4

@sbeaumont

Helping test the v8 status: It is not deployed yet because we still need to do internal QA. @connor and I will be working on this one for the next few days. I will send you a message privately for information on how to do an early access preview, if you are comfortable with using unstable “edge” features.

Reading pins:

> I’d need a solid way to monitor the changing live data so I can confidently determine the bot is ready for the next command

We agree that this is a pain point and are researching solutions. This week, @connor and I will experiment with a “return types” concept, such as returning a coordinate when a move_abs completes or an integer when read_pin completes.

One idea we have considered is to add an MQTT channel that broadcasts the return value (eg: the coordinate node mentioned above) of an RPC on a channel that references the original correlation ID of the RPC. Under this scheme, bot/device_123/rpc_response/87601920-1ff2-11e9-ab14-d663bd873d93 might broadcast a coordinate node to let you know the result of a move_abs command. There are still some legacy support issues we will need to tackle before we could roll out such a feature. I can’t make too many guesses on a timeframe for this feature yet.

Identifying mounted tools: It is currently a manual process, and as you correctly noted, it can’t be a final solution to the problem. We’ve seen some folks researching RFID tags for tools, but that is still in the research phase. Using binary codes on the UTM mount is an interesting solution to the problem that I will forward to the hardware folks.

Eliminating REST API usage / out-of-band responses: We’re working on a possible solution for this, where all data is managed via MQTT. The feature is complete but it has not been QAed and the documentation is still in the early phases.