[FARMWARE] Mother's Little Helper (MLH)


#41

my 2c …

I understand what @Klimbim is asking, but I am with @connor on this. FB project shall not be viewed like an appliance (i.e. dishwasher). It is an open system that assumes certain technical expertise from the user. Yes, the error message displayed by FB looks cryptic, but it provides enough information to conclude that the problem is elsewhere. Asking FB to chaise for the source of these problems doesn’t worth the effort.

Remember the troubleshooting wizard in MS Vista?
Internet-Connection-Troubleshooting-Wizard
I don’t know a single person whom it helped.

Thank you,
Eugene


#42

Warning, this post will be long.
To help add some context on how open the Internet architecture is…I created a crude SysML model that hopefully will help shed some light on the topic. I don’t claim to be an expert in IP networks, and please remember that all models are wrong, some are useful. With that opener, here we go!

Step 1. Download this zip file (It contains an HTML report of the model and the model file itself if you’re model saavy or wanna try out Cameo Systems Modeler)
IP Network Infrastructure.zip (2.6 MB)

Step 2. Open the IP Network Infrastructure.html file in your favorite web browser.

Step 3. Navigate to the “Internet” structure diagram (see below diagram)

Primer for reading these
black diamonds mean "has a"
white diamonds mean "uses a"
white triangle means “is a”

The internet “has a” DNS (this translates names to network addresses) and “has many” IP Networks. The IP network uses the DNS to translate names to network addresses (this is one possible source of your issue…but not likely…more on this later. An IPS “is a” (type of) IP Network. Each IP Network “has one to many” nodes. Servers, Clients, and Routers are all “types of” Node.

Step 4. Open the “Domain Naming System” structural diagram.
The Domain Naming System (DNS) translates user-readable names to their network address. For example, forum.farmbot.org translates to the network address of 104.131.20.102. Your Farmbot’s network address is provided by your Internet Service Provider (ISP) (or quite possibly your local network). Now referencing the “Internet” structure diagram, realize that there is a series of routers provided by your home network and all of the ISP networks between your Farmbot and the server. This is where the “traceroute” utility can be of some service. This utility will ping every router between it’s node and the node it’s trying to reach, displaying the results. A dead ping indicates a node is down. Most Linux-based OS come with the tracert utility de facto. I can’t remember if Farmbot OS is Linux-based or not.

** Step 6.** Open the “Home Network” Structure diagram.
This diagram lays out the structure in a prototypical internet-connected home network. Most people’s these days connect to their homenetwork via their Wireless Access Point (WAP), which then will reach the router, then the ISP, then the Internet. For a large majority of users, the WAP and router are combined (i.e. ‘wireless router’). Generally speaking, these consumer grade WAPs are terrible, terrible, terrible and are usually the source of most internet issues. Since the Farmbot can’t really do anything about a bad WAP, this is the main thrust behind @connor’s and @etcipnja’s opinion that this is not in FB OS’s scope. I happen to agree, but I also feel that some basica troubleshooting information could be provided in the logs if these base-level networking tools are included in the OS. In the event of any network issue, it could try to resolve the DNS, ping a server, and provide a tracert output. This information could help the user (us Farmbotters) more quickly isolate issues. You still will need to do these same steps from another node in your network to completely isolate the issue.

Step 7. Open the behavior diagram “Exchange Information”. I’ve included a full version screen grab below, but the fonts may be a little small. In this one, you can see all of the interactions that have to take place between your FB and the FB server. If anyone of these nodes in the chain fail, so does your communication, and ultimately whatever function your FB is performing that requires information or service from the FB server.
This is also what makes troubleshooting, isolating, and fixing issues hard and impossible respectively. Per @etcipnja’s point, it’s an open-system. The FB cannot exert any control, nor influence, on the network. Per @connor’s point, it can be pretty difficult for FB to isolate issues automatically given the sheer diversity of network configurations out there, and there are plenty of dedicated network analysis tools out there that do exactly this. In other words, it’s out of scope. As stated above, I think there are a few ‘freebies’ available to the FB dev team that they could integrate in, if they get the free cycles to do it (i.e. no other major things going on to fix/develop).


If you have made it this far, I thank you for reading. You have earned 17 internet points. If you have any questions, let me know and I’ll try to point you in the right direction. I agree with everyone else’s assessment: you very likely have a WAP issue (external interference, your WAP is drunk again, or you live near an airport (like me)). FWIW, I have commercial grade gear and I still have issues, albeit much less than I did with the consumer grade gear. This is also why I have made the decision to hard wire my FB to my network. This does involve burying cable and installing a grounding rod to protect your network gear, so let me know if this option is attractive to you. OK, it’s whiskey time on Friday! Cheers all!

Fuzzy


Farmbot network troubleshooting
#43

Finally! Those are the last 17 i needed!

Really though thank you so much for this. I think we should add this post to the FAQ.

Farmbot OS is Linux-based, but very stripped down. It’s pretty much just the kernel and Farmbot OS runtime and a tiny subset of other packages. (wpa-supplicant for example). Logging the output of commands could be helpful, but unfortunately due to the nature of how Farmbot requires network to send logs, it can be hard to read those logs. I’m working on building a system that will allow Configurator to keep more than the last-most recent log that i think should help users see what caused their bot to go offline.

This is my preferred way to keep Farmbot (and any other networked device really) online. That being said i have quite a few Farmbots online via just WiFi, and i never really have any issues.
I’d really love to see your buried network setup though. This might encourage users to set this up themselves.

Update

Not to derail this already hijacked thread any further but i wrote up a quick FAQ document to hopefully help users troubleshoot why their bot disconnects. You can find it here


#44

@connor Please, please give us the way to ssh to it! I’ve spent so much time reverse engineering it via farmware-log window - you won’t believe.

Eugene


#45

I’ll add information about this to the FAQ.

Update

@etcipnja I’ve updated that FAQ.


#46

Shouldn´t we split the thread and continue this online network detection discussion in a separate thread, pls?! I do not like to high jack ets thread…

Quick question: what about trying to ping the WiFi Access station and see wether pings get through? This would then rather pin point to the bad WAN connection? Moreover you could save the RSSI value with the last logs and even keep logging it?
I definitely agree with @etcipnja on his standpoint that farmbot should not be used to debug a faulty network / internet installation / system but nevertheless it could help people to make their farmbot work…


#47

@etcipnja please help:

Warn CeleryScript: #PID<0.7382.1> died: {:badarg, {GenServer, :call, [Farmbot.HTTP, {:request, :get, " https://raw.githubusercontent.com/etcipnja/MLH/master/MLH/manifest.json", “”, [], []}, :infinity]}} 0, 0, 0.17 Aug 21, 9:11am
2

Busy Installing farmware from https://raw.githubusercontent.com/etcipnja/MLH/master/MLH/manifest.json. 0, 0, 0.17 Aug 21, 9:11am

I am trying to install MLH after the farmware was deleted due to OA update to new OS. My plants really need water so I am quite stressed :wink:


#48

@Klimbim

The error says: Your Farmbot tried to get a copy of MLH farmware from external resource (GitHub) owned by Microsoft. During this process it failed.

The farmware did not get a chance to get executed because it was not yet installed.

So, you may ask FB support on why it failed to fetch the farmware. If I’d guess it is because FB OS did not have access to the network during this time. May be the same DNS issue.

Thank you,
Eugene


#49

This cannot be really. If the bot is online and I am working in it but only the farmware always makes problems there`s something wrong in the code. I don´t know which information the team needs to further debug…?


#50

@connor

Please let us know if Farmware can be responsible for the problem during installing it

Warn CeleryScript: #PID<0.7382.1> died: {:badarg, {GenServer, :call, [Farmbot.HTTP, {:request, :get, " https://raw.githubusercontent.com/etcipnja/MLH/master/MLH/manifest.json", “”, [], []}, :infinity]}} 0, 0, 0.17 Aug 21, 9:11am


#51

Looks to me like there is a space in the url.

" https://raw.githubusercontent.com/etcipnja/MLH/master/MLH/manifest.json"
 ^ this guy right here

but for what it’s worth that string should probably be better sanitized somewhere in the stack i suppose.

UPDATE: Yep just tried it with and without the space. It caused a failure when there was a space.


#52

@connor @Klimbim

So the conclusion is:

  • @Klimbim made a honest mistake by adding an extra space
  • FB developers will add a trimming of whitespaces from user input in the future releases to avoid this problem in the future

correct?


#53

Sounds correct to me! :slight_smile:


#54

I already tried several times and cut the rest on the right side when I copied the link. However it seems I had missed the space on the left part, which I haven´t obviously seen by the way - I do not have the coder eye :wink:
Nevertheless when I tried a third time, copying the link from another source it worked.

So the proposed solution seems suited and should avoid further inquiries from users with “copy errors” :wink:

Thanks for supporting @etcipnja and @connor


#55

Glad to be of assistance! I’ll update the model as time permits and community needs develop! I will try to edit this post to have pictures a bit later. I know they’ll be helpful. Ans, sorry for the delay, my family is preparing for the arrival a baby girl in October so my time has been spent getting all-of-the-things ready.

As for the hard install, the broad strokes are as follows:

Material/Tools

  • Outdoor/Burial-rated conduit - flexible PVC is recommended. Needed for data and power cable external legs of the install (at a minimum).
  • Outdoor rated junction boxen (4x) - essential for code and installation flexibility. Two each for both data and power (one at the FB end, one at the house end).
  • Outdoor rated GFCI outlet - I used a single gang fixture, which gives me some future capacity for other things that may need power in my back yard.
  • Outdoor rated Data cable - Essential for the final outdoor leg to the Farmbot. I used Cat 6 for technology reasons.
  • Water hose that can be buried. Much easier than PVC, but less permanent. I had an old hose laying around, so I used it for simplicity.
  • 8ft copper ground rod with fitting collar (for the ground wire)
  • fence post driver
  • 2lb hammer

Recommended

  • A separate 15amp or 20 amp circuit breaker. GFCI rated is optional, depending on code
  • Data cable surge suppressor. Strongly recommended to protect your communication infrastructure from lighting.

Order of burial: electricity, data, water. Why? The electrical line is the most dangerous and there are codes regulating burial depth. It’s also the thing that is designed to be in the ground the longest. The hose I chose for the water line will likely deteriorate the quickest, and when it starts leaking, I want to know about it. Having it closer to the surface allows me to replace it without disturbing the other lines.

  1. Cut a trench about 12" (0.33m) deep. I buried water, power, and data in this trench. The depth allowed for separating the three lines for electrical safety code reasons. Your local code may vary. Mine required 6" (0.15m) of separation and a buried depth of at least 6" (0.15m). I also recommend saving your turf (if you have any) on a tarp so that you may reduce the time of the ‘bad haircut’ visual effect after you’re complete.

  2. Lay in the 3 conductor (power, return, earth ground) power line. I used a direct-bury line, which removed the need for conduit for the majority of the run. You’ll still need conduit from getting out of the ground to your electrical box. You also can use conduit. I recommend a ground-rated flexible PVC conduit. Perform a conductivity test. Cover with 6" of soil.

  3. Prepare the data cable for burial. They do have outdoor rated, cabling with a burial-rated jacket. The cost is similar to buying a non-outdoor version and a protective conduit. For my configuration, I chose to put Cat6 riser cable inside of a element proof, desiged-to-be-buried conduit. This allowed me the ability to easily replace the data cable in the future as technology changes occur. I ran the conduit to a jucntio box, and then used outdoor rated Cat6 for the final leg (from the junction box, through the x-axis cable carrier, to the Farmbot box. In this step, you’ll need to pull the data cable through the conduit when it’s possible to stretch out the conduit and reduce the number of elbows.

  4. Lay in the data cable. Run a continuity test. Fill with dirt.

  5. Lay in the water line. Test the water line. Cover with 2" of soil.

  6. Replace the turf.

  7. Connect all of the conduits/service lines to their respective junction boxen/endpoints. I STRONGLY recommend using a GFCI outlet. Most electrical code will require this.

  8. Install an 8’ copper ground rod. A fence post driver, a 2lb hammer, and moist soil are the best tools/conditions for this. I recommend driving the rod in at a 30° angle as this will encourage the rod to either go over or under any rocks it finds on the way.

  9. Connect the ground to all chassis and surge equipment. (I strongly recommend a surge protector for your data cable). Be sure to check for ground loops as well as these can be dangerous.

optional
I ensured that any ground exit points were well guarded from lawn equipment by using bricks.


#56

@etcipnja:
Today I wanted to execute the farmware (it´s still installed ;)) and I encoutered the following error:

Error CS Failed: [execute_script] - “Farmware failed” 0.2, 0, 0.5 Oct 9, 9:20pm
2

Warn|#Farmware<MLH(2.4.16)> completed with exit status: 1|0.2, 0, 0.5|Oct 9, 9:20pm|
|2

Busy|Beginning execution of #Farmware<MLH(2.4.16)>|

I rebootet the bot and got the same.

Any idea what is happening here? The error log does not tell me anything… :confused:


#57

sorry, my bad - it was syntax problem. Now it shall be better. Update to 2.4.17

I am having problems with my FB - can’t verify it now. Also I am busy with another project lately - so I won’t be able to provide timely support :frowning:


#58

Works again, thanks very much!

Which problems do you face?

Cheers


#59

Hey Guys,

since the new WebServer/API Version (v7) - MLH does not work for me anymore. Is it just me, because of something else, or has someone similar problems atm?
Trying to narrow down the problem, so feedback is very appreciated :slight_smile:


#60

Hello, this is sad.
It is pretty much winter in MD. On top of the fact that my FB died I am about to winterize it anyway. So I have no way to work on the fix right now. Sorry. I hope community will pick up this project. I’ll try to answer questions if they arise.

I am not giving up with FB yet. This was a lot of fun! See you next season.

Thank you,
Eugene