I don’t know. I’ve been experiencing a few ‘ghosts in the machine’ myself lately. Yesterday, all of my 3rd party Farmwares disappeared. I had to reinstall all of them. I did not trigger any updates. I addition to this, the Farmbot itself was rapidly disconnecting and reconnecting. I still haven’t figured that one out yet.
@connor Any ideas on what would cause Farmware to ‘disappear’?
What version of FarmbotOS are you on? I believe that particular issue should be fixed as of 6.4.7 if i remember correctly. they were not being uninstalled or wiped out, they just weren’t being shown due to a bug when reconnecting. It seems like whatever is causing your rapid “disconnect and reconnect” may have caused that (hopefully fixed) farmware bug. I’d be interested to see what is causing your disconnect and reconnect however. When you say rapid, how rapid? Any notable logs?
I can confirm 6.4.7 of FarmbotOS. The periodicity of the connection up and down was in seconds, and I attempted to record diagnostics a few times , however, I needed to be connected in order to tell it to record diagnostics, which was a bit chicken-egg It may have been successful at least once, so feel free to pour over them if you have the ability. My FB is current offline as I told it to use the wired connection and it’s yet to nab a DHCP address. I have yet to read on how to get it to ‘reset’ and reenter setup. Would you be kind enough to point me in the correct direction? Once back online, I can help troubleshoot some more.
I’ve moved this to a new topic as not to distract the original post.
As to “resetting” Farmbot, the intended behaviour is that it will only reset when it detects there is a unsolvable problem (Farmbot can’t boot, bad username and password, invalid Wifi Password etc)
There is a known problem where if you select “wired connection” during configuration, Farmbot will never reset as long as your username and password are correct. So my guess is that Farmbot was at one point connected to the web services, so it knows the password is correct and since you are using a wired connection, it thinks that is ok also. I don’t think there is an easy way of manually resetting the bot, and this is a good use case for such a thing. I will add it to our feature list. I do suspect that unplugging and replugging Farmbot from mains will at least get you back up and running. If you upgrade to the latest release (6.4.10 as of writing), there is a new feature that will place a circular log dump available to download. This is to help us (Farmbot developers) figure out what is going on with a particular setup.
The power cycling worked. I was able to get 6.4.10 to install. The Farmware I had previously installed is again ‘wiped’. See screen shot below. So far, I haven’t noticed the rapid cycling re/connection behavior. I’ll walk the FB through a few sequences more today and report back.
Sad to hear about Farmware being “wiped” i’m sure they still exist on the filesystem, they just aren’t being discovered for some reason. This initial implementation of “Farmware” has quite a few things broken that i have been working on refactoring pretty much non stop over the last 2 weeks.
Keep me updated if you see FarmBot disconnecting and reconnecting rapidly again.
Connor this is understood but not agreed to only put it onto the shoulder of the “user”. As it depends pretty much how farmbot tries to establish connections to different services, at least the particular thing that causes a complete shutdown (for 30 mins.(!) in my example) should be logged so that the user can work on it. How could I distinguish between something which is bad on my network or maybe services from farmbot which are not available?!
I really do not get why you are so strongly defending not logging what exact reason caused the bot to disconnect or go into “sleep” or factory reset mode?
Today for example, I found the bot disconnected since yesterday evening. That means that it was not pingeable in the network. However, after I powered on / off, it came up and booted into the same network. This was different to other times, when it did not come up again and I needed to connect to farmbots network and configure it again.
I do not experience any problem with other services so it is strongly evident that farmbot does something in a different way.
Its like a child @connor if you do not know why its crying its hard to help
6.4.9 Fix bug causing FarmBot not to reconnect to WiFi.
There was a bug in the version of wpa_supplicant Farmbot OS uses relating to the Raspberry Pi’s wifi chip driver. (brcmf_mac) along with FarmBot not being properly notified when disconnected. (that is disconnected from the access point)
The exact reason is being logged, it will just never make it to the backend services since FarmBot is disconnected. This is why i built the feature to capture all previous logs.
There is also a bug in that after a reconnect, those logs will never actually make it back to the backend services. Rick and I have been working on a solution for this, but it is actually a really complex issue not just limited to logging.
Farmbot has a huge disadvantage that a user can only indirectly communicate with Farmbot. (IE it needs to be connected to the network, and a user needs to be connected to a network etc) It is similar to ChromeCast, or similar devices. Unfortunately we don’t have the same development team that Google has
Could that explain:
a) the disconnect that I just needed to cycle power on the bot?
b) to reconnect the farmbots wlan and reconfigure it?
Which is not a problem for the moment being, I could diassemble the SD card for such an event. I appreciate that you are working on a longterm solution!
No problem @connor, we are working on this as a community. Its just frustrating if you cannot do anything to fix those disconnect problems and thus if you really (sadly just) want to use the FB to water while you´re away…
Is this solved through the SSH connection? Could we connect to it in the local lan and look at the syslog?
Could you point me to the services that farmbot is using so I could ping them regularly and see if such a service is not available when the bot disconnects? Would that point me towards the right direction?
Thanks for you help (finally)
PS: Maybe we should split the topic and continue in the disconnection thread…?
This will allow you to see logs, but only while the session is connected. See the document for more information.
If you have access to a machine that can mount a ext4 filesystem, you can certainly see a dump of logs. the file will be called debug_logs.sqlite3
This is a bug that should be fixed in 6.4.10 which will actually be rereleased as 6.4.11 because 6.4.10 had some sort of bug in it that didn’t get caught during QA. I’m working on getting this out hopefully today.
Maybe it helps, some more debugging information. In the pictures below you can see that a raspi in my lan pinged the bot every minute, same as with the google dns servers. You can see when no response was received anymore (value of 0). Before you can see some pings which were considered to not reach it already. What you can see as well is that my latency of pinging the google server does not change and I have internet connection through the router at any time.
The second picture shows the RSSI value received from the bot being stable…
The last log entries from the bot are:
Busy Syncing 0.4, 0, 0.83 Sep 10, 11:23pm
Busy Syncing 0.4, 0, 0.83 Sep 10, 10:54pm
Busy Syncing 0.4, 0, 0.83 Sep 10, 10:24pm
So even that it was pingeable after 11:23 pm up to 12:10 in the morning no more logs were saved.
around 11:50 on the RSSI graph, the connection gets pretty bad. It’s possible that this caused a timeout, and a disconnect. It also looks like after about 12:10, the RSSI graph stops. To me it seems like FarmBot disconnected from your wifi access point at this point.