Farm events constantly missed


#1

Hi,

I am experiencing a constant problem that Farm Events not being fired. I have two events scheduled every day at 9pm and at 11:50pm.

Sometimes they fired but mostly not. Today I was waiting for the event watching the logs. It did not fire, the FB was connected all the time.

This is very problematic because I can not be sure that watering started or not. So I have to check it manually every day.

Besides asking you to take a look on it I have a proposal:
It would be cool if event had a “validity period”. If event was not fired for some legit reason on the time scheduled - it will be tried again every minute during the validity period.

In fact people don’t care if watering starts 9pm or 9:30pm. But they care that it started today. It is unfortunate that due to temporal issues the watering is skipped completely.

Just to reiterate. In my case I do not see a legit reason why it did not fire.

Thank you,
Eugene


#2

Very interesting. I have a number of FarmEvents that run every day throughout the day and they’ve never missed more than ~5 or so minutes. I’d be interested to see the contents of your event. Is it a repeating sequence? a one time Regimen?


#3

They are FarmEvents scheduled every day on a repeat.

Eugene


#4

I have reoccurring sequences on very similar times (but in the AM). Farmbot should have no problem executing those events. Is it both sequences that are missed? is there any pattern you’ve noticed? Does your bot have a stable internet connection? is it possible the events are firing, but the bot isn’t connected while they fire?

This is a known bug that i’m working on a fix for. Currently logs while disconnected don’t get dispatched to the Web Services.


#5

Both events were missed. The screenshot above was taken in the morning next day. So you see the last log was on 7:25pm. I don’t see the pattern when it works and when it doesn’t.
BTW: I think I saw other people complained about the same on the forum …

Here is my wifi connection now. I have no reasons to believe that it was different yesterday.
24 PM

Here is the quote from my original post:

Today I was waiting for the event watching the logs. It did not fire, the FB was connected all the time.

… I was waiting for the event by monitoring farmbot web page …


#6

Ah sorry for missing that part. When you say you were

waiting for the event watching the logs

Were you actually watching the bot itself?

Your connection looks good to me. I see no reason why Farmbot would be missing events.


#7

I was literally sitting next to the bot watching it and the logs simultaneously :slight_smile:


#8

Thanks! I’m looking into this right now and we are discussing possible solutions to a very related issue where Farmbot will be able to tell a user when+why something did/didn’t execute.

As for your issue at hand i am trying to recreate your events as exactly as i can to see if it is something with the logic of scheduling. I have noticed a bit of clock skew due to the RPI not having a built in clock that has caused a FarmEvent to be off by a margin of about 5-10 minutes, but it’s never caused an event to actually be missed. This is currently what i suspect to be at play.


#9

Okay i’ve set up a dedicated test for this one.

I’ll keep an eye on it over the next couple days to see what is going on.


#10

@connor

Today my bot rebooted itself out of blue several times. Every time before reboot I see the following line

“Time seems to be set”

As far as I can see in logs it happened 3 times and every time before reboot this line was displayed.
Last reboot finished Aug 7, 8:51pm , at 9pm sharp the Farm event supposed to fire. It did not happen.
hope it helps.

You said you have logs in the DB - please take a look.

Thank you,
Eugene


#11

This is very strange. I don’t think your bot is rebooting, but the Farmbot software is crashing and restarting. I’m having a read through your logs. don’t think the log that caused the issue will have been dispatched tho since it has the possibility of containing sensitive data. For what it’s worth i setup the same times as your screenshot from above and the one scheduled to run at 9:00 executed right on time. I will investigate more…


#12

@connor
It is 9:02pm EST. I was closely watching the bot both personally and via the web. To make sure that bot is alive I turned lights on and off 2 min before 9pm (you may see it in the logs)

“Water All” event did not fire @9pm.

What is interesting - I saw it disappeared from scheduled events on the web for today. Like if it was fired.
I guess I need to install your debug build. But before that I’ll reschedule all events. May be it glitched after sw version upgrades

Thank you,
Eugene


#13

Unfortunately, the web app tracks FarmEvents separately to FarmbotOS. It disappearing from the api doesn’t actually mean it fired. We are working on a solution to this.

Rather than installing the debug build, i’ve released a beta image that does the same thing, but doesn’t require reflashing your sdcard.


#14

@connor here is the update on the subject: after I deleted all events and created them again the problem disappeared.


#15

Hey glad to hear that. For what it’s worth, i been spending a lot of time bettering error detection and collection so I can better help debug these issues.


#16

This is still a problem. I had the exact same thing happen as of 3/17/2019.