Are the farmbot servers struggling a bit? It took me several attempts to log in (about 10-15) and the connection goes offline a lot:
Same here. Did they sell a whole bunch of them recently?
@aronrubin @ClosedCircuit Thanks for letting me know. I am investigating now. My guess is that nice gardening weather + large shipment of Express bots last months = unexpected server spike.
@ClosedCircuit It looks like your client was actually rate limited for making too many requests. My initial fix was to set a throttle rate at which “noisy” clients would be temporarily blocked from the server. It appears I set that limit too low and am deploying a fix to address the issue.
Please let me know if you still have trouble accessing the server and I can take a look.
@RickCarlino - Spoke too soon. I can’t sync (sync error button - if I click on it, I have a pop-up that says “sync failed”):
I also have the following error message at the top of the page:
… and I’m locked out again:
@ClosedCircuit It seems like your bot (or user account) is reading a lot of sensor readings all at once and is being throttled as a result. It does not appear to be a system-wide issue and my Express bot is doing fine it seems. The last outage was because someone was running an API script with a bug in it (was spamming our server on a slow endpoint) so I had to be more aggressive about how we throttle requests.
Can you tell me more about your sensor usage? Are you running any API automation / scripting tools (there’s nothing wrong with that, by the way- I’m happy to help you adjust your scripts to meet throttle requirements if that’s the case).
I would need to learn more about what you are trying to do to understand your situation or if there is a bug that is causing the frontend to make too many HTTP requests.
@RickCarlino. It’s a pretty off-shelf configuration and I am not running any API automation / scripting tools. I am not sure why it would do a lot of sensor readings.
At the moment, it’s supposed to run 2 sequences per day (watering and soil sensor) and that’s it.
I’ll re-flash the sd card and see if that helps.
I have re-flashed the SD card and the firmware and restarted several times. The Farmbot tells me that there are sync errors but pressing the sync button doesn’t help. I cannot control the Farmbot either.
The logs show a ton of “Error preloading…”. Several per minute. I don’t know why.
—Addendum—
Having switched off the Farmbot for a couple of hours, the same issue was still there when I rebooted the machine. Scrolling through the log, the following message also appeared in the log (in the middle of many “Error preloading…” messages:
Hi @ClosedCircuit . . Yup, I think we need more capacity on the server-side !
Look closely and you’ll see the Same 429 HTTP response status : https://httpstatuses.com/429
Thanks @jsimmonds. That’s consistent with the log but I don’t understand why there should be all these requests. I haven’t done anything fancy with the machine. It’s bog-standard Genesis XL 1.5, no APIs or scripting, latest FBOS and firmware. I have a few sequences, but again, nothing fancy.
As you say, maybe it’s a capacity issue on the server side.
Aahhh, Ooops, you’re right ! 429 is a rate-limiting response per-client
Your bot is only doing 8 requests per minute though, which doesn’t sound a big deal . . and also, it’s doing preloading requests which you have no control over !
I think there’s a logic error in the server rate-limiting code.
I just want to chime in and say I am still facing the “retry later” error. Thanks.
@waldo The “Try Later” error usually means that your bot performed a large number of requests and was temporarily rate limited. We are getting a higher-than-usual volume of these reports, which leads me to believe the problem is on our end. I will un-throttle your device manually for now and take a deeper look on Monday. Thanks for the outage report.