Went to prepare the FarmBot for a presentation today and experienced some issues I’ve never seen before. It seems to be working better now but here’s a list of the issues I was having.
The FarmBot constantly keeps trying to set itself home even though its at the home location.
FarmBot keeps going online and offline.
FarmBot keeps locking itself when I would send a sequence command, then lock itself and then slowly go back to home as if I was calibrating the axis.
FarmBot keeps logging that “firmware needs to be opened”
And a whole other slew of errors.
* Sync Error
* Preload EconnError
* Movement failed
* Disconnected from PingPong channel
I have a picture of the log messages. I was having some troubles uploading the picture so I put a google drive link. I’ve tried all of the basic troubleshooting issues such as restarting and flashing the OS but I’m stumped at this point.
All those 429 status codes (“Too many requests”) could be an indicator that the device got blocked somehow. But I’m not sure if that even exists for http requests.
Did you try to boot the device with the arduino/Farmduino unplugged?
Also try to flash the newest OS version.
If nothing helps we could ask RickCarlino for assistance, he might know what exactly prevents the device from syncing.
Let me confirm the version update. before I flashed to 7.0.1 I did check to see if there was an update and it said ‘up-to-date’
There is a chance also that the district possibly blocked the FarmBot, just thought of that now. I’m going to contact them and see if they blocked the FamBot.
Hey there Christophe - we’ve been experiencing very similar issues with our system. Currently using the latest OS 8, but issues were occurring even before updating. Similar scenario - our STEMbot (coined this term as our system is used for design based STEM education) behaves as if trying to set itself at home, even though its already there. As well, it will no longer “Set Zero” for any axes.
The result is that my middle school teacher is at a standstill for continuing to use our STEMbot in her classes. After going through every reboot, update, re-flash, etc. that might resolve the issue, we’re at a loss on next steps. About to order a new Farmduino and Pi, though the web site indicates all of these are Sold Out - so we can’t even replace parts to see if that would resolve the issue.
Any suggestions would be much appreciated by the teacher and her students.
Yup I’ve made some good progress. First, I did confirm with our district if they blocked the FarmBot, they did not. I re-flashed once more and then re-updated to 8.0.3.
I’m receiving fewer errors. I no longer am receiving “firmware needs to be opened” and any other errors that I was previously getting except for yesterday, where I got three Preloading errors. (See photo)
The FarmBot is not trying to send itself home when its already at the 0,0,0 location anymore and It is not dropping online and offline anymore, pretty much all of the issues I was having seemed to resolve themselves with a re-flash.
I am a little confused. The day of the issues, I tried reflashing twice with no luck. I made the post and then checked it yesterday and everything is good to go. Not sure why the initial re-flashing didn’t do the job the first two times.
@Cjaramillogrows The error above appears to be an HTTP throttle (DDoS protection). To see if that’s the case, I cleared out all throttles on the server. I checked your logs just now and it looks like your device did eventually manage to sync.
I am a little confused. The day of the issues, I tried reflashing twice with no luck. I made the post and then checked it yesterday and everything is good to go.
When a device is throttled, the server sets an expiration time. I have a hunch the throttle was still in effect when you flashed the first few times. It’s just a hunch though.
Your logs appear healthy right now- is the device working again?
I have an update and I’m baffled. Our FarmBot has good days and bad days, Tuesday I noticed it was connecting and disconnecting randomly, same as when I originally posted this. Checked back today and everything is working just peachy.
I am still getting “Error preloading. [error: :econnrefused, error: :econnrefused, error: :econnrefused]” every once in a while but besides that, no other errors.
You mentioned DDoS protection. Were you saying that the DDoS protection is on your side, or our side? Is it possible that our district is throttling the FarmBot because its constantly trying to ping back and forth between the district network and the FarmBot and maybe that’s why its bring throttled?
Were you saying that the DDoS protection is on your side, or our side?
It is on our side and it is unfortunate, but necessary. If you have a very poor wifi connection, it will get triggered because the device will try to re-connect too often. Initiating a connection is resource intensive on our end (we need to re-run authorization), plus there is an incurred memory overhead on our server because zombie connections use up memory. We had a bad week a few months ago where there was increased traffic + a poorly written farmware that was leaking sockets. It crashed the message broker and we ended up with the rate limiter we have today as a result.
Is it possible that our district is throttling the FarmBot because its constantly trying to ping back and forth between the district network and the FarmBot and maybe that’s why its bring throttled?
It’s possible. Some sysadmins don’t like long-lived TCP connections and will randomly close them. I’ve seen this at a few larger organizations.
I will take a look at your device right now and see if I find anything noticeable.
I wanted to provide an update on the FarmBot here. I’ve gotten in contact with our district to see if they are closing the TCP Connection, Our onsite IT said he will get back to me once he can look into it.
In regards to issues, everything seems fine once more. I had one issue earlier while testing, where the FarmBot went offline in the middle of a sequence, came back online and then tried to home the Y-axis back to the zero position. It ended up assuming that its last recorded coordinates for the Z and X-axis were their home position. After I synced and then Zeroed the X, Y, Z-axis I re-ran the sequence and everything went just fine.
We have an ethernet jack here near the FarmBot, I’m planning on hard-lining from this point forward once the district opens access to that jack.
Thanks for all the help with this. I will keep in touch if anything changes.
Possible Solutions?
I wanted to discuss more details on possible solutions if this is a district network to FB server issue.
We have another contract upcoming with an elementary school where we are going to purchase 2 FarmBot Express units. If a hardline is needed to solve this issue that does create a problem since these units will be outside and will not be able to be hard-lined. Of course, this might also be related to the TCP connection which could be a simple fix If the district is aware of the connection being closed and allows the connection to stay active for an indefinite length of time.
@whitecaps, you mentioned a Wifi booster might work? I wanted to see if you have tested this yourself.
P.S. Should I start a new thread for this second half of the conversation?
Thanks for the updates @Cjaramillogrows and sorry to hear it’s still not back 100%.
I am curious to see what your IT department says. If the hardline behaves the same way as the WiFi, then the culprit is almost certainly the local network. If the hardline performs better than the WiFi, I think local network conditions would be less likely (and the WiFi booster may be a viable option)
I appreciate the amount of detail in your updates, and it’s fine for us to continue the discussion here in this thread.
I had to use a wifi booster with my old router because the signal wasn’t strong enough. It worked really well. In fact I make go back to using it even with my stronger router as there are still some trouble spots. It was super easy to set up.This is the one I have NETGEAR Net-EX3110-100NAS AC750 WiFi Range Extender https://www.amazon.com/dp/B077K3L7DQ/ref=cm_sw_r_cp_apa_i_8K3PDbRSCZ4C7
Well, hopefully we will hear back soon. (our district IT takes forever to reply… lol) At least whatever we find can help FarmBot implantation within districts.
Two questions Rick,
Is there a way I can obtain the diagnostic recordings from the FarmBot web app?
With the FarmBot, every time I establish a network connection (say after the FarmBots been unplugged for a day) and connect to our district network, is the FarmBot getting a new IP address? I was talking with our onsite IT and he mentioned its possible that If we have our district assign the FarmBot its own registered IP address then the TCP connection wouldn’t get terminated. That is IF this problem is because of the connection getting terminated on our end due to connection time.
