Farmbot offline, logs show cyclical pattern

Troubleshooting
1

Aloha Folks,

Our Farmbot mysteriously went offline yesterday. I tried cycling the power but this did not help. Today I re-imaged the OS to v 7.0.1 and turning auto-update off in the chance that 7.0.2 was breaking something, and I noticed connectivity briefly after the initial setup wizard but it then disconnects and the logs show a cyclic pattern.

I have made no recent changes to anything myself and the Farmbot is tethered with Ethernet.We are on a school network and I have put in a request with the network division to see if anything has changed higher up, but no word on that.

The log shows as follows:

Busy	Initializing Firmware.	0, 0, 0	Apr 9, 9:56am
Busy	Firmware not initialized yet. Waiting for R88 message.	0, 0, 0	Apr 9, 9:56am
Success	Got ARDUINO STARTUP COMPLETE. UART is up.	-1, -1, -1	Apr 9, 9:56am
Success	Successful Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:56am
Busy	Beginning Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:56am
Success	DNS resolution successful	-1, -1, -1	Apr 9, 9:56am
Success	Interface eth0 is up.	-1, -1, -1	Apr 9, 9:56am
Busy	Waiting for interface eth0 up.	-1, -1, -1	Apr 9, 9:56am
Info	Starting Networking	-1, -1, -1	Apr 9, 9:56am
Busy	Configuring Farmbot.	-1, -1, -1	Apr 9, 9:56am
Success	detected target UART: /dev/ttyACM0	-1, -1, -1	Apr 9, 9:56am
Busy	Checking /root/boot	-1, -1, -1	Apr 9, 9:56am
Success	Got ARDUINO STARTUP COMPLETE. UART is up.	-1, -1, -1	Apr 9, 9:55am
Success	Successful Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:55am
Busy	Beginning Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:55am
Success	DNS resolution successful	-1, -1, -1	Apr 9, 9:55am
Success	Interface eth0 is up.	-1, -1, -1	Apr 9, 9:55am
Busy	Waiting for interface eth0 up.	-1, -1, -1	Apr 9, 9:55am
Info	Starting Networking	-1, -1, -1	Apr 9, 9:55am
Busy	Configuring Farmbot.	-1, -1, -1	Apr 9, 9:55am
Success	detected target UART: /dev/ttyACM0	-1, -1, -1	Apr 9, 9:55am
Busy	Checking /root/boot	-1, -1, -1	Apr 9, 9:55am
Success	Got ARDUINO STARTUP COMPLETE. UART is up.	-1, -1, -1	Apr 9, 9:54am
Success	Successful Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:54am
Busy	Beginning Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:54am
Success	DNS resolution successful	-1, -1, -1	Apr 9, 9:54am
Success	Interface eth0 is up.	-1, -1, -1	Apr 9, 9:54am
Busy	Waiting for interface eth0 up.	-1, -1, -1	Apr 9, 9:54am
Info	Starting Networking	-1, -1, -1	Apr 9, 9:54am
Busy	Configuring Farmbot.	-1, -1, -1	Apr 9, 9:54am
Success	detected target UART: /dev/ttyACM0	-1, -1, -1	Apr 9, 9:54am
Busy	Checking /root/boot	-1, -1, -1	Apr 9, 9:54am
Success	Got ARDUINO STARTUP COMPLETE. UART is up.	-1, -1, -1	Apr 9, 9:52am
Success	Successful Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:52am
Busy	Beginning Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:52am
Success	DNS resolution successful	-1, -1, -1	Apr 9, 9:52am
Success	Interface eth0 is up.	-1, -1, -1	Apr 9, 9:52am
Busy	Waiting for interface eth0 up.	-1, -1, -1	Apr 9, 9:52am
Info	Starting Networking	-1, -1, -1	Apr 9, 9:52am
Busy	Configuring Farmbot.	-1, -1, -1	Apr 9, 9:52am
Success	detected target UART: /dev/ttyACM0	-1, -1, -1	Apr 9, 9:52am
Busy	Checking /root/boot	-1, -1, -1	Apr 9, 9:52am
Success	Got ARDUINO STARTUP COMPLETE. UART is up.	-1, -1, -1	Apr 9, 9:49am
Success	Successful Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:49am
Busy	Beginning Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:49am
Success	DNS resolution successful	-1, -1, -1	Apr 9, 9:49am
Success	Interface eth0 is up.	-1, -1, -1	Apr 9, 9:48am
Busy	Waiting for interface eth0 up.	-1, -1, -1	Apr 9, 9:48am
Info	Starting Networking	-1, -1, -1	Apr 9, 9:48am
Busy	Configuring Farmbot.	-1, -1, -1	Apr 9, 9:48am
Success	detected target UART: /dev/ttyACM0	-1, -1, -1	Apr 9, 9:48am
Busy	Checking /root/boot	-1, -1, -1	Apr 9, 9:48am
Busy	Initializing Firmware.	-1, -1, -1	Apr 9, 9:48am
Busy	Firmware not initialized yet. Waiting for R88 message.	-1, -1, -1	Apr 9, 9:48am
Info	API is the source of truth for Farmbot OS configs. Downloading data.	-1, -1, -1	Apr 9, 9:48am
Info	API is source of truth for fw configs.	-1, -1, -1	Apr 9, 9:48am
Warn	Firmware is being stubbed.	-1, -1, -1	Apr 9, 9:48am
Busy	Initializing Firmware.	-1, -1, -1	Apr 9, 9:48am
Busy	Firmware not initialized yet. Waiting for R88 message.	-1, -1, -1	Apr 9, 9:48am
Warn	Firmware is being stubbed.	-1, -1, -1	Apr 9, 9:48am
Success	Successful Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:48am
Busy	Beginning Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:48am
Busy	Initializing Firmware.	-1, -1, -1	Apr 9, 9:48am
Busy	Firmware not initialized yet. Waiting for R88 message.	-1, -1, -1	Apr 9, 9:48am
Warn	Firmware is being stubbed.	-1, -1, -1	Apr 9, 9:48am
Success	Successful Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:48am
Busy	Beginning Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:48am
Busy	Initializing Firmware.	-1, -1, -1	Apr 9, 9:47am
Busy	Firmware not initialized yet. Waiting for R88 message.	-1, -1, -1	Apr 9, 9:47am
Warn	Firmware is being stubbed.	-1, -1, -1	Apr 9, 9:47am
Success	Successful Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:47am
Busy	Beginning Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:47am
Busy	Initializing Firmware.	-1, -1, -1	Apr 9, 9:47am
Busy	Firmware not initialized yet. Waiting for R88 message.	-1, -1, -1	Apr 9, 9:47am
Warn	Firmware is being stubbed.	-1, -1, -1	Apr 9, 9:47am
Success	Successful Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:47am
Busy	Beginning Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:47am
Busy	Initializing Firmware.	-1, -1, -1	Apr 9, 9:47am
Busy	Firmware not initialized yet. Waiting for R88 message.	-1, -1, -1	Apr 9, 9:47am
Busy	Initializing Firmware.	-1, -1, -1	Apr 9, 9:46am
Busy	Firmware not initialized yet. Waiting for R88 message.	-1, -1, -1	Apr 9, 9:46am
Warn	Firmware is being stubbed.	-1, -1, -1	Apr 9, 9:46am
Success	Successful Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:46am
Busy	Beginning Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:46am
Busy	Initializing Firmware.	-1, -1, -1	Apr 9, 9:46am
Busy	Firmware not initialized yet. Waiting for R88 message.	-1, -1, -1	Apr 9, 9:46am
Warn	Firmware is being stubbed.	-1, -1, -1	Apr 9, 9:46am
Success	Successful Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:46am
Busy	Beginning Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:46am
Info	API is the source of truth for Farmbot OS configs. Downloading data.	-1, -1, -1	Apr 9, 9:46am
Info	API is source of truth for fw configs.	-1, -1, -1	Apr 9, 9:46am
Busy	Initializing Firmware.	-1, -1, -1	Apr 9, 9:46am
Busy	Firmware not initialized yet. Waiting for R88 message.	-1, -1, -1	Apr 9, 9:46am
Warn	Firmware is being stubbed.	-1, -1, -1	Apr 9, 9:46am
Success	Successful Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:46am
Busy	Beginning Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:46am
Busy	Firmware not initialized yet. Waiting for R88 message.	-1, -1, -1	Apr 9, 9:46am
Warn	Firmware is being stubbed.	-1, -1, -1	Apr 9, 9:46am
Success	Successful Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:46am
Busy	Beginning Bootstrap authorization: farm.bot@holualoa.k12.hi.us - https://my.farm.bot	-1, -1, -1	Apr 9, 9:46am

An image of the log for readabilitty:

cyclic%20log
cyclic log.jpg1914Ɨ2033 964 KB

2

Same here, started with this exact problem: Farmbot disconnected after latest OTA OS update

Then tried a fresh 7.0.2 image on the sd card and hit this same loop.

Installed 7.0.1, unchecked the ā€œFARMBOT OS AUTO UPDATEā€ option, yet it still grabbed an OTA update and then started this whole loop again.

Farmbot has been offline now for days.

Untitled
Untitled.png1920Ɨ942 106 KB

1 Like
3

Aloha @newt80ns,

Iā€™m not glad your Farmbot is offline but am glad that someone else can confirm the same issue showing that it is not an isolated incident.

Sending out an S.O.S. to development team:
@Gabriel
@RickCarlino
@roryaronson

4

Same problem here.
Farmbot seems to be stuck in ā€œInitializing firmwareā€ and ā€œWaiting for firmware idleā€. Flashed two times, didnt solved the problem. Please check if you have a software issue here.

We run on FarmOS 7.0.1 and have a Genesis 1.4.

5

Hi folks, we are investigating the issue. We will keep you posted on the fix shortly. Sorry for the downtime.

1 Like
6

Can you try booting Farmbot without the Arduino data cable plugged in please? This should hopefully cause Farmbot to boot into a limo, but stable, connected mode and will help determine what system is failing

7

hereā€™s what I got yesterday, doing a fresh flash with 7.0.1 and auto updates turned off. got it to complete the one move command, went back into a sync loop as soon as it rebooted with the update.

farmbot%20logs%20print%20screen
farmbot logs print screen.png1927Ɨ1075 137 KB

hereā€™s what I got today, booting things back up with the Arduino USB data unplugged

farmbot%20logs%201207
farmbot logs 1207.png1915Ɨ1071 152 KB

for information, we run our v1.4 FarmBot connected with ethernet cable, was running stable with 7.0.1

8

Logs after booting with the Farmduino disconnected:

arduino_disconnect
arduino_disconnect.jpg1574Ɨ1897 1.11 MB

9

Hi Guys,

Iā€™ve got a very similar issue. Iā€™ll have a go with the arduino disconnected.

My farmbot says that itā€™s up and running but it wonā€™t move anywhere.

Iā€™ve managed to plant one seed so far. Very frustrating!

Thanks,
Mark

10

Hi All,

Tried re-flashing with 7.0.1 and the arduino disconnected. It automatically updated to 7.0.2. and it said that this was successful. I turned off auto update and it appeared to sync.

Tried to reset it and reflash / configure it with 7.0.1. It still did the over the air update upon connection and continues to be stuck in the firmware initialisation cycle like other users have described.

The connection light behaviour is correct during the initial startup phase with a solid blue light. The green light then blinks, the blue light goes out and then doesnā€™t come back on. The system is then disconnected from the internet and the error logs stop coming through on the web interface.

Have any changes been made in 7.0.2 that would change how the device is appearing on a network / is accessing the internet? It was a mission to get it working in the first place as I work in a school with our internet filtering company needing to create exceptions. Given that thereā€™s no blue light, this may be the issue.

I have power cycled it several times and re-configured several times.

Thanks,
Mark

11

We have the same issue on our schoolā€™s Farmbot. Here are some of our logs.

image
image.png652Ɨ641 22.7 KB

12

@RickCarlino

Please can you simply remove the 7.0.2 OTA update so we can use the FarmBots until you find the problem? There is currently no way to flash an older version as the bots are ignoring the ā€˜Do not update OSā€™ flag in the configuration and grabbing the OTA update anyway.

I imagine you can still leave the full 7.0.2 image up for those wanting to try it, but the OTA update is killing a number of bots right now.

3 Likes
13

Weā€™ve identified this as not being a problem with the farmbot os itself, but it is a connectivity issue, probably due to network latency. Currently investing possible solutions.

14

@connor Excellent, so, can you pull the 7.0.2 OTA until itā€™s resolved? :slight_smile:

15

This effects previous releases as well, so pulling it would not solve anything.

16

For our school Farmbot, we were running great for about a month before the 7.0.2 OTA update. As the one log shows, I had a working connection with a reflash and reconfig using 7.0.1, that was capable of connecting to and moving the robot. It worked for as long as it took the bot to update itself. As soon as it rebooted with 7.0.2 I lost all functionality.

Seems like pulling 7.0.2 OTA update would get us running full speed ahead again.

17

I have same issue and i was working myself with the farmbot - playing with sequences, watering plants etc, when all of sudden fb stopped being connected. It disconnected exactly after it auto updated its os to 7.0.2. Again - i experienced this live, literally while I was working with my fb. So i firmly believe this is does not have anything to do with network. I think reverting back to previous versions of sw in case some of the clients (farmbots) experience business interrupting errors in production is standard procedure and should be available to everyone. Why we are not even trying this / disable OTA update and revert back to 7.0.1 and see if that helps? Its much better approach than letting farmbots down for days in productionā€¦

18

Iā€™ve published farmbot os 7.0.3. It should hopefully contain a fix for the problem.

19

Hi Guys,

I am quite a long distance away from my farmbot. Could someone please test this fix ASAP? I donā€™t really want to travel all that way to have it fail on me again.

Also, how do I get the farmbot back to the configuration stage if I cannot connect to it to reset it?

Thanks,
Mark

20

Flash the SD with 7.0.3 on another system.
Restart FB with 7.0.3

1 Like