Trying to improve the weed Detection function（Using Deep Learing）

Software Software Development
1

Hello, we are a small team of farmbot builders sitting in China.
We are trying to combine image processing and deep learning technology with fasrmbot, and we intend to try to improve the original weed detecition function first, and then proceed to develop crop recognition, growth condition, pest and disease functions one after another after successful testing.
During this period, we encountered some problems：
We are trying to combine image processing and deep learning technology with fasrmbot, and we intend to try to improve the original weed recognition function first, and then proceed to develop crop recognition, growth condition, pest and disease functions one after another after successful testing.
During this period, we encountered some problems.
Since we have never used Elixir, we want to use other languages (e.g. python) to implement communication between web front-end and devices (e.g. cameras), but we are not sure how Farmbot os interacts with web front-end, can you explain the communication flow between them roughly or provide a flowchart? Thanks.
Another question is that currently the photo result of the web page must be refreshed to see, can we get the photo taken in real time and display it on the page, is it related to the rerender mechanism of the React component? Or can we get the video stream from the endoscope camera directly, which will help us a lot to get the training data and do image processing.
Hope to get your replies to help our development work.
Best Regards. :smile:

2 Likes
2

Hi @SK.LIN ! Software Documentation for this is at Web App | FarmBot Software Development Documentation

My summary : Web App API ( REST-ful JSON ) manages bot Resources for the Account and MQTT channels manage Device Commands, Responses and logs etc.

A Python 3 interface published by the FarmBot Developers also has documentation at FarmBot Software Development | FarmBot Software Development Documentation.

[edit]
The FarmBot Inc.-supplied and supported code for take_photo is this Python 3 code here.

2 Likes
3

@SK.LIN That sounds like an interesting project. I would take a look at the documentation mentioned in @jsimmonds message above.

The best way to do this is to use a wrapper library like FarmBot.js or FarmBot.Py. Please see the developer docs for more information.

On Chrome and firefox, the photo will eventually show up. It takes some time for post-processing (~30 seconds).

This is currently not possible.

4

We just spent the Spring Festival holiday, sorry for the late reply.
First of all, thanks all for your replies, which gave us some new ideas. We will try something new next and ask for advice again if we have any questions, thanks again.

1 Like
5

@SK.LIN Not possible via any FarmBot-supported API currently existing. But is possible if you work closer to Buildroot Linux ( that FBOS runs on ) and the Raspberry Pi hardware :slight_smile:

6

I just got a used Farmbot Genesis (FB) from another school. I’m also thinking about using machine learning (ML) to improve weed detection. Currently, I’ve trained a small model (MobileNet2) to spot common weeds in our area, and it’s working well. To integrate it with the FB, I’ve figured out these steps:

  1. Scan the garden and get images using Farmbot OS and the Python API take_photo.py.
  2. Process the image using ML to obtain a list of coordinates of the weeds.
  3. Call an API or function (I’m not sure which one) to move the weed tool to the point and eliminate the weed.

Now, I have a couple of questions:

  1. Should we perform this via a Web API or by using Farmbot OS Python functions?
  2. Which APIs or Farmbot OS functions should I look at for step 3?

Thanks

1 Like