Farmbot servers overloaded?

Are the farmbot servers struggling a bit? It took me several attempts to log in (about 10-15) and the connection goes offline a lot:

image

…and I also get messages like:

The latest message has just answered my question… :wink:

Same here. Did they sell a whole bunch of them recently?

@aronrubin @ClosedCircuit Thanks for letting me know. I am investigating now. My guess is that nice gardening weather + large shipment of Express bots last months = unexpected server spike.

2 Likes

Hi @RickCarlino. For your information, the issue is happening again now (5:35pm California Time).

image

@ClosedCircuit It looks like your client was actually rate limited for making too many requests. My initial fix was to set a throttle rate at which “noisy” clients would be temporarily blocked from the server. It appears I set that limit too low and am deploying a fix to address the issue.

Please let me know if you still have trouble accessing the server and I can take a look.

1 Like

I’m in. Thanks a lot @RickCarlino!

1 Like

@RickCarlino - Spoke too soon. I can’t sync (sync error button - if I click on it, I have a pop-up that says “sync failed”):

image

I also have the following error message at the top of the page:

image

… and I’m locked out again:

image

@ClosedCircuit It seems like your bot (or user account) is reading a lot of sensor readings all at once and is being throttled as a result. It does not appear to be a system-wide issue and my Express bot is doing fine it seems. The last outage was because someone was running an API script with a bug in it (was spamming our server on a slow endpoint) so I had to be more aggressive about how we throttle requests.

Can you tell me more about your sensor usage? Are you running any API automation / scripting tools (there’s nothing wrong with that, by the way- I’m happy to help you adjust your scripts to meet throttle requirements if that’s the case).

I would need to learn more about what you are trying to do to understand your situation or if there is a bug that is causing the frontend to make too many HTTP requests.

@RickCarlino. It’s a pretty off-shelf configuration and I am not running any API automation / scripting tools. I am not sure why it would do a lot of sensor readings.

At the moment, it’s supposed to run 2 sequences per day (watering and soil sensor) and that’s it.

I’ll re-flash the sd card and see if that helps.

I have re-flashed the SD card and the firmware and restarted several times. The Farmbot tells me that there are sync errors but pressing the sync button doesn’t help. I cannot control the Farmbot either.


The logs show a ton of “Error preloading…”. Several per minute. I don’t know why.

—Addendum—

Having switched off the Farmbot for a couple of hours, the same issue was still there when I rebooted the machine. Scrolling through the log, the following message also appeared in the log (in the middle of many “Error preloading…” messages:

Hi @ClosedCircuit . . Yup, I think we need more capacity on the server-side !

Look closely and you’ll see the Same 429 HTTP response status : https://httpstatuses.com/429

Thanks @jsimmonds. That’s consistent with the log but I don’t understand why there should be all these requests. I haven’t done anything fancy with the machine. It’s bog-standard Genesis XL 1.5, no APIs or scripting, latest FBOS and firmware. I have a few sequences, but again, nothing fancy.

As you say, maybe it’s a capacity issue on the server side.

Aahhh, Ooops, you’re right ! 429 is a rate-limiting response per-client :confused:

Your bot is only doing 8 requests per minute though, which doesn’t sound a big deal . . and also, it’s doing preloading requests which you have no control over !
I think there’s a logic error in the server rate-limiting code.

1 Like

I just want to chime in and say I am still facing the “retry later” error. Thanks.

@waldo The “Try Later” error usually means that your bot performed a large number of requests and was temporarily rate limited. We are getting a higher-than-usual volume of these reports, which leads me to believe the problem is on our end. I will un-throttle your device manually for now and take a deeper look on Monday. Thanks for the outage report.

1 Like