FarmBot just shuts down/goes offline after a few hours

I am trying to root cause a fail where the FarmBot seems to shut down.
Symptoms:
After about an hour or three, we are unable to connect to our Bot.
The two external LED’s go dark.
The internal RazPi red led goes on, flashes for a bit, then goes off. After a few seconds it repeats.
De-energizing (no power for 10 sec) has no effect.
Flashing and re-configuring the FBOS seems to work to get it back to nominal, but so far it repeats each time.
I have swapped (and ruled out) RazPi’s.
I would like to know some steps to rule out the Farmduino.
We have had two hot days (100+ F), so maybe heat-related.

And the plot thickens…

After the FB “died” yesterday, I decided to be a bit more meticulous in the steps to figure out wtf is going on…
so, this am i did the following:

  1. Observed that only the red LED on the RazPi was on. It had VERY faint flickering - almost un-noticeable.
  2. Observed that the red LED on the Farmduino was on and steady.
  3. Observed that none of the external LED’s were on.
  4. Removed power (at the Farmduino) for 3 minuets)
    a) No change
  5. Pressed and held the “Reset” button on the Farmduino for 30 sec.
    a) RazPi red LED turned off, on, off, on… (seemed cyclical)
  6. Disconnected wired LAN cable - no change
  7. Removed power for 10 sec, pressed and held reset button as I re-applied power.
    a) RazPi red LED turned off, on, off, on… (seemed cyclical)
  8. Disconnected Farmduino - repeated steps - no change
  9. Removed SD, re-flashed FBOS to 7.0.1
  10. Set up Mobile Hotspot
  11. Configurator started (farmbot-xxx SSID broadcast)
  12. Configured to wired LAN settings
  13. FarmBot is ALIVE!
  14. Kicked off the standard watering sequence - so far, so good…

Thoughts:
I am starting to suspect that either there is a marginal problem with either the RazPi and/or Farmduino that is corrupting the FBOS
or…
Something else is corrupting the FBOS.

I cant understand why I cant manually force a factory reset and get back to the Configurator without having to re-flash the FBOS…

Attaching a log dump…

OvoFarmBotLogs_2019_06_12.txt (16.8 KB)

Everything after 10:33 am is part of the events that led to it going offline.

Adding @Marc
Maybe you can give this some traction, please.
Not a show-stopper, since I have a way to get it to at least 1 water cycle a day, but it would be great to root-cause and resolve…

Tony, can you help me out by creating a short video of exactly what is happening? Speak on the video so that we can see exactly what is happening.

How hot exactly did it get? Are you able to Jerry-rig some shades for the FarmBot electronics box. Can you also check the powersupply and measure the voltage at the Farmduino? (Should be 24V)

WHEN THE EXTERNAL LED’s GO DARK [SPECIFICALLY THE BLUE LED] THE NETWORK CONNECTION IS GOING OFFLINE AND YOUR PORTS ARE LIKELY BLOCKED

SEE THIS TROUBLESHOOTING DOCUMENT FOR MORE INFORMATION: https://software.farm.bot/docs/status-leds

image

STEP 0 – Unplug the FarmBot and power everything down then power everything up and Restart using the FarmBot Web App. Start the camera rolling and clearly show which version of the FarmBot OS you are running.

Step 1 - Repeat the procedures that you mention in your post and try to get it on camera:
After about an hour or three, we are unable to connect to our Bot.
The two external LED’s go dark.
The internal RazPi red led goes on, flashes for a bit, then goes off. After a few seconds it repeats.
De-energizing (no power for 10 sec) has no effect.
Flashing and re-configuring the FBOS seems to work to get it back to nominal, but so far it repeats each time.

It sounds to me like your IT folks are playing games with you or network has some automatic sequence that is changing the port settings on your network.

See this link for more information on Blocked Ports:

FarmBot requires the following ports to be open on your network:

  • PORT: 5672 AMQP
  • PORT: 80 HTTP(S)
  • PORT: 443 HTTP(S)
  • PORT: 3002 WebSockets

Marc