I’ll run the cable (+15 ft) from the Farmbot to a (dumb) switch, then to the router. To me, a dumb switch merely received packets and sends them out to everything connected to it. That means the packet will get “regenerated” electrically.
If I can get a good connection, I can get the IP address and then run a traceroute to see the loss of packets.
-B
Sounds good @bcatanzaro please let me know how it goes and also when the device is accessible via public internet so that I may remotely inspect logs.
OK… we are well connected now according to pings. The cable goes from the Farmbot directly to an existing switch which is hard-wired to the router. The time delay of the ping indicates to me that we have great connectivity.
Here’s the dialog box showing the F/W revision and local IP address.
However, the web interface keeps oscillating between Synced and Sync unknown
Here’s the diagram of connectivity:
and top level logs
All controls are currently offline. I’m not sure why. It is not in an E-Stop state. It should be activated.
This is caused by latency, packet loss or packet filtering of outbound traffic to the message broker data center (VA, USA).
AMQP messages travel over the internet and back. They don’t take a direct path over the LAN.
Although you can see your FarmBot from within your LAN, that doesn’t mean you have great connectivity to the AMQP broker. Those packets need to travel all the way to the AWS data center in Virginia on a long lived (never closing) outbound TCP port 5672. Furthermore, AMQP requires a certain level of “liveness” to connections. It has a ping/pong mechanism that is handled at protocol layer (not a traditional ping like you are doing). It kicks devices off of the server if they are unable to reply to this pong message in time, as you can see from your “flickering” UI.
Are you absolutely certain there is no software that would be closing long lived TCP connection? We’ve seen some security software that does not like long-lived, encrypted AMQP traffic. The other common problem is latency (not latency within the LAN but rather latency to the AMQP server). Are you able to temporarily disable security settings or try configurating the Pi from a different internet connection altogether (eg: someone else’s WiFi hotspot or a 4G hotspot?)
I am leaving the office for the day but will be back at it tomorrow. There are other possible factors that I can investigate tomorrow as well. I’m sorry this is taking so long. All of the issues you have mentioned look like a typical local network issue (we see this precise behavior at schools and enterprises during onboarding).
The time delay of the ping indicates to me that we have great connectivity.
What I meant by this is that within our LAN, pings are good. This means our switches and cables have good wiring and that nobody on our LAN is hogging all the B/W.
Are you able to temporarily disable security settings or try configurating the Pi from a different internet connection altogether (eg: someone else’s WiFi hotspot or a 4G hotspot?)
Not sure how to accomplish this without accessing the Configurator. Since this weekend, we have been unable to get the Configurator to appear. Although we have the tech for WiFi hotspotting, this means we cannot configure the Farmbot to interface to a WiFi radio.
Are you absolutely certain there is no software that would be closing long lived TCP connection?
I did not specifically enable / disable something to disable long lived TCP connections. Signals are going over a standard cable modem connection to the Internet. I had previously set up a Python script to measure bandwidth every 30 min. We routinely get +200 Mb/sec download, but no home internet in our area is 100% reliable.
The LAN router is a WNDR4500 which interfaces to the cable modem / router. It has typical
I have to look at the cable modem in more detail to see what software it is running.
I need a little more guidance with regards to what I’m looking for in the router configuration…
A freshly flashed FBOS will always start the Configurator WiFi AP.
The AP SSID can take a couple of minutes to show up.
Is there any other WiFi AP or 2.4GHz ISM-band transmitter nearby that might be “jamming” ?
Configurator uses only 2.4GHz band.
( I’ve not seen this but it has been reported in this forum IIRC )
Yes there is ( this was already quoted earlier in this thread )
https://software.farm.bot/docs/for-it-security-professionals
Not sure how that helps.
Just confirming : You’re still troubleshooting the loss ( after 10 minutes ) of connectivity to WebApp or MQTT broker ?
@jsimmonds I’m mostly concerned about the 97% ping loss with AMQP/MQTT (see: the connectivity panel diagram). I think the Web App part is fine.
If you are using a stock, consumer-grade home router, you probably don’t need to investigate this. This was my hunch because your device is exhibiting typical signs of network blockage (the kind we see at schools and large orgs with strict security policies).
As John mentioned, we do have the document for security professionals (I’m repeating myself here in case other users / future users have found this topic via search). I’m happy to answer any questions you may have about that document.
Self hosting is possible for advanced user. It will be time consuming and it will not be a simple process. It also might not work, but we can at least try it. If you are able to get it running, it would be a good way to eliminate certain problems since the traffic stays inside the LAN. If the problem is something else, such as 2.4Ghz interference, it will make no difference.
Self hosting requires:
The most common mistakes people make when self hosting is:
MQTT_HOST when the server’s IP address is actually 10.11.1.235).Now that I have gotten the disclaimers out of the way, the setup instructions for a self-hosted server can be found here.
Once you have the self hosted server set up, you will need to:
Please let me know if this works.
I don’t have a full FarmBot set up in the garden, but on my desk I have created a “custom” Pi image and am running a local WebApp until the garden is deployed. My Internet connection is unreliable (just satellite/long haul wifi), so using remote servers won’t work. Plus I don’t like SaaS in general…
I have been able to get both the Pi image and the WebApp running locally without too much pain, but I’m a sysadmin. If you have deployed docker in the past, it is reasonably straightforward. I have been able to successfully update the Pi image and WebApp without losing data. I’m running the WebApp in a KVM (virtual machine) locally.
I use Debian instead of Ubuntu. I copied and slightly modified the FarmBot WebApp install docs for Debian. Perhaps this may save time for Debianistas:
Happy hacking,
-Jeff
Oh that’s interesting to see @jebba. I’ve heard Docker support had improved in the last 4 years. Thanks for sharing.
Is your “Pi image” running on virtual or a real Pi ? Just curious. Amazing what can run on top of what these days ! Don’t want to “rabbit-hole” this thread, but I’ve been retired from IT for a few years now.
Is your “Pi image” running on virtual or a real Pi?
It is the real Pi that comes with the FarmBot Genesis XL 1.5 (Pi 3). To be clear, I’m just running the standard FarmBot OS on it, not raspbian or anything like that. So my setup with the camera is the same as when the FarmBot is deployed, it’s just sitting on my desk, not in a garden (yet).
@jebba Would you be willing to start a discussion thread to share the process you used to get a server running on a Pi? This might be a bit off-topic for the curent thread, but a lot of users ask about this, so it would be nice to have a reference.
@bcatanzaro Are you going to try a self-hosted setup? Please let me know if you are still having issues and I would be happy to continue the troubleshooting process.
Sorry, I wasn’t clear. I have an x86_64 Debian Buster server in a KVM running the WebApp. I’m not running that on a Pi. I’m using The Pi just as you ship it, in the FarmBot enclosure, running the FarmBot OS (though with my build).
Are you going to try a self-hosted setup?
Not yet. I have a computer that is running Ubuntu LTS 18.x. Not quite prepared to run this yet
Connectivity using Wired Connection
Was able to program regimens through the Web interface and syncing was pretty consistent. Please tell us what you saw in log files circa June 16 afternoon. Could you see traffic / connectivity was improved?
Configurator
Still unable to get the Configurator to engage. I wish that there was a way a user could trigger the Configurator to occur. I figured re-flashing the FarmOS would cause the Configurator to engage; it did not. Power cycling did not either.
Firewall / Port Forwarding
Our LAN routers don’t have any Port Fowarding or Blocking of Services enabled. However, our router / modem to the cable ISP may block ports. Tried ( https://canyouseeme.org/ ) to search my IP address for open ports. Nothing was open (including Port 80 / HTTP and other common ports). Not sure if that means it is blocking the ports Farmbot needs or if the port scanning service is not doing what I expect.
Spoke to a buddy at my ISP. My ISP has many ports blocked. I can configure Port Forwarding, but this requires a fixed IP address for the device that needs port forwarding. The docs say:
FarmBot does not require a fixed IP address. DHCP is the preferred method of assigning an IP to a device. Due to internal network configuration variation, some organizations may need to assign a static IP to a device, although this is less common.
I don’t see anything in the Configurator docs that help me establish a fixed IP address.
My initial attempt to do this ( today ) on an RPi4B has failed 
But remembering Bob Dylan’s lyrics from Love Minus Zero/No Limit :
She knows there’s no success like failure And that failure’s no success at all
I’m not stopping here 
I was in this step
# Install application specific Ruby dependencies
sudo docker-compose run web bundle install
and this happened . . ( Note that naughty x86 business 
)
That’s fine. As you may have noticed from some comments in the forum, self-hosting is not always easy to set up. We can always save this as a last resort if there’s no other option.
Unfortunately we only track signal strength for WiFi. It looks like your device switched to Ethernet around 6/13. If you want to try leaving the device online with ethernet and sending me a message on the forum, I can take a deeper look when it is online.
You can access configurator by visiting the device’s IP address over HTTP. So, if your bot’s IP is 1.2.3.4, you would visit http://1.2.3.4. You can get the bot’s IP on the configurator page, but you need to have an MQTT connection to the bot to view this information, and unfortunately a lack of MQTT stability is the exact problem we are debugging:
Since you probably can’t see that pane, you could instead use a networking tool like the Fing App and scan your network for Raspberry Pis.
To address your original statement, the feature you are requesting does exist and you can trigger configurator from the UI. The problem is, the UI tells the device to factory reset itself using MQTT/AMQP, and your device does not have a good connection to that:
Another option which I would not recommend is the “network reset timer”. This is a timer that will factory reset the device after X minutes of lost connectivity. This is mostly used by people that move their farmbot through different WiFi networks frequently. We receive a large number of support requests from customers who forgot the timer was active and become confused when they notice their bot has factory reset itself during a power or internet outtage.
This tool is only useful for troubleshooting software that exposes TCP ports (eg: someone who runs an MQTT server inside their home LAN). In the case of FarmBot, we are using the ports as a client rather than acting as a server. It would make sense that there are no open ports visible, assuming you do not run any servers on your LAN.
Please see my earlier comment about visiting FarmBot’s local IP on the LAN via web browser.
We have the ping tool in the connectivity widget. Last time I looked, it appeared to be dropping pings pretty regularly >50% of the time.
You don’t need port forwarding to run a FarmBot. The only exception is if you are self hosting a server and want to share the server with people outside your LAN.
A good question to ask since it seems you have easy access to ISP staff would be to see if they have any ideas on why your connectivity (or possibly latency) is spiking so high for AMQP over SSL. The document I sent earlier (“FarmBot for IT Professionals”) might give them some ideas.
You will need to setup your router to support this. Once your router is set up to support static IPs, you would go into the “ADVANCED SETTINGS” section of the network config menu of configurator:
I don’t think a static IP will help, but if you are knowledgeable in network administration, you could try it.
So to recap:
If we don’t make any traction soon, perhaps we can hop on a phone call. I am in US central time and work typical business hours. You can send me your phone number via PM, or LinkedIn since we are already connected on there.
@jsimmonds Good luck! This is something a lot of folks have asked for. Perhaps you could start a new thread, since I imagine others would like to know your progress.
