This version will be released later today. Please see http://os.farm.bot for updates.
I’ve been having FarmBot Message Broker connectivity issues for the past 24 hours, consistent across reboots. Internet connectivity is A-OK (see below). Could my issues be related to the new OS release?
1 Like
and I just realized this is a Beta release, so probably not.
@fuzzynickel My guess is that this is not related to the new OS release, but I can take a deeper look to be sure tomorrow morning.
2 Likes
@fuzzynickel Are you absolutely certain there were no changes to your local network in the past few days? Is your device in a corporate / school setting? If so, have you contacted IT administrators to verify that there were no policy changes recently?:
- We’re not seeing a spike in these sort of reports (number of devices that are showing this issue on the production server is in the single digits)
- On the server side (our end), the logs are simply saying that the client connection was closed unexpectedly.
- On the device side, when SSHed in, the device is saying the the connection timed out with no response (but usually after a brief connection to the server).
I am going to email our AMQP vendor to see if they have any ideas. In the mean time, an interesting experiment would be to try unplugging the Pi from the electronics box and re-configurating on a different ISP to rule out LAN configuration issues.
Other things I tried:
- I thought that perhaps this could be an issue with how we set our RabbitMQ heartbeat.
- Tried flashing an experimental build with a heartbeat of 60
- Tried a heartbeat of 0.
- Neither seemed to help, so I am unsure if this is a problem with the AMQP config. For reference, the server default heartbeat is 10s. If the device does not send a heartbeat every (10/2) seconds, the client is assumed to be timed out. This is a problem for some connections.
Other things I can try:
- I’ve been meaning to try switching from AMQP to MQTT (due mostly to the heartbeat issues a small percentage of users face). This might not be an option. More research is needed.
- Contact our AMQP for outside support, as mentioned previously.
Please let me know if you have any other information or notice anything different.
2 Likes
@RickCarlino Posting some notes from our side conversation.
- I’m the IT admin for my network (Ubiquiti UniFi stack for the curious) and I have not changed any policies since I created the network 2 years ago.
- I have updated the firmware on the gateway/router and the WAPs. (~ 2 weeks ago)
- I have changed the DNS to 2 new PiHoles (~3 weeks ago).
I’ll have to get creative to try your suggestion as I’m hardware limited and I can’t take down my network during business hours (I am fortunate to be able to work remotely). I do have an idea to isolate the fb from UniFi stack and go straight to the ISP gateway (AT&T).
As for your experiments, I did not notice a connectivity change, other than the log started capturing a lot more connection events, and it looks like the Pi / FBOS has been in a cycle of resetting itself owing to loss of connectivity since your firmware changes yesterday. Hopefully the logs will shed some more light. I sincerely appreciate the hard work you’ve been putting in for me, and others over the past few weeks with the AMQP/MQTT brokerage issues!
I also admin a UniFi setup at work. You should be able to VLAN the FB and put it outside the firewall. Separately, I recall having to do something funky to let SIP phones through and I wonder if something similar is needed for MQTT.
1 Like
After a quick look, I see UniFi itself uses MQTT for communication from controller to components.
1 Like
oh, now that is interesting…could the Unfi stack be intercepting FB’s MQTT messages? The VLAN should technically solve that, yes?
Before you VLAN it off, you should try adding a new firewall rule for MQTT. From the internet:
Next lets configure your NoT firewall rules. For me, my NoT needs to be able to communicate via MQTT with my MQTT server, so I’m going to make a rule called Allow NoT to MQTT, select accept, then under source I’ll select the NoT network, then under destination select address/port group, and I’ll add a new group under ip addresses that I’ll call Home Assistant with my specific home assistant server IP address since that is where my MQTT server lives, and under port group I’ll add a new group called MQTT ports that will contain the two common MQTT ports which are 1883 for non secure MQTT and 8883 for secure MQTT.
Thanks! I’ll give that a go later today.
You might as well do AMQP as well which is 5671 and 5672
https://www.rabbitmq.com/uri-spec.html
1 Like
For clarification: The FBOS stack does not currently use MQTT. The browser uses WebSocket MQTT (not true MQTT) and the bot uses AMQP.