API / HTTP Requests: DNS Implementation in the Farmbot, nxdomain issues, UDP vs. TCP

Hi there,

My sequences sometimes fail due to DNS problems. Seeing errors like these in the logs:
{:error, :nxdomain}

The root cause is probably my unstable Wifi network since my Farmbot is behind a repeater this might be not ideal… :face_with_monocle:

Anyway, I’m thinking of many users may could have similar issues, since DNS queries are made with UDP. Lost packets will never be returned to the Farmbot.

Would it may be possible to implement a new setting to switch DNS to TCP based DNS. I found this article it probably can be done easily in the resolve.conf: domain name system - how to configure Linux to use TCP for DNS queries? - Server Fault

Another thing would be to enable some sort of DNS caching… But this could be not ideal for DNS Loadbalancing…

By the way: Is it possible to do a packet capture on the raspberry it self?
Today I made a trace on my router, and I saw (I’m using plant group location based sequences), on each iteration a new DNS query is made. The router replied to all of them, but shortly before I stopped the trace, I got the “nxdomain” error. It must have been some sort of fatal error since the whole sequence stopped in the first quarter of processing 70 plants.

Errors:

Wireshark printscreen:

Does that Wireshark screen shot capture those FBOS errors at 22:04 ?

I’m not convinced that your router DNS is to blame for these :nxdomain errors (?)
It seems to reliably return DNS query responses in under 10ms.
Are you thinking that some UDP response packets are not arriving over the Wi-Fi connection to FBOS ?

Potentially, yes. You will need to add tcpdump to the FBOS Linux build (Buildroot Linux forum should have the ‘how to’ details), then make your custom farmbot_system_rpi3-portable-1.23.1-farmbot.1 package and use that in your FBOS build.
It should also be possible (if you can get a compatible tcpdump executable + required SO libraries) to sftp those to /tmp and run them from there. This requires you to have a custom FBOS build which include ssh support (not hard).

You might need to balance the effort of setting up DNS over TCP/HTTPS versus running a wired Ethernet connect to the bot. (Easy to do with Ethernet over Powerline devices)

Edit:
@farmS1m the FBOS low-level DNS code uses Erlang OTP modules. You can examine their code to see what timers might be used before that :nxdomain failure is asserted.

Edit:
@farmS1m I looked in the FBOS code but did not find “Error during Metadata creation - See logs [ . . . ]” ?

Yes the trace has been started at 21:46, and run for 19 minutes. I stopped the trace shortly after the issue occured.

Yes thats what I’m thinking of. Therefore I might will either try to setup a rpi image with tcpdump, or will consider to use a Powerline :neutral_face:

Thats correct it is a custom log message from my code:

plant = variable("Plant")
plant_id = plant.id
plant_url = "/api/points/" .. plant_id 
watered_last_amount = tonumber(env("watered_last_amount"))

-- toast(plant_id)
-- toast(plant_url)

result = api({
    method = "put",
    url = plant_url,
    body = {
        meta = {
            watered_last = local_time("date"),
            watered_last_timestamp = tostring(os.time()),
            watered_last_amount = watered_last_amount
        }
    }
})

if result then
    -- toast("Metadata creation ok", "debug")
else
    send_message("error", "Error during Metadata creation - See logs for details", "email")
end


api_result = api({method = "GET", url = plant_url})
result = json.encode(api_result)
-- send_message("info", result)

From the trace it is hard to say at least it looks like the FB did a request after the last DNS query seen in the trace. Probably the next dns query done from FB is not included in the trace, as It never got transported to the router. I don’t know.

Thank you so much for your helpful ideas! :smiley:

1 Like

Ask Wireshark to display absolute timestamps . . assume your router is nicely NTP synchronized (?)

Maybe another capture which completely covers a “:nxdomain” occurrence will show us bigger clues.

Edit: P.S. I also :two_hearts: Wireshark :slight_smile:

1 Like

Getting one step closer. The setting to display absolute timestamps in Wireshark has helped a lot! I was able to pin it to 22:04 (UTC 20:04).

It looks like the FB received a new DHCP offer at that time. Probably because the Wi-Fi Adapter lost its signal and recovered from it a few minutes later. Unfortunately my trace was made with a host filter (of the FB IP), therefore I do not see the DHCP request from the FB with origin IP 0.0.0.0.

I think this is proof enough, my Wi-Fi is still :poop: (Even I already have installed a repeater, the signal strength fluctuates between 36 - 84%! This is quite bad, but think the shutter blinds have some impact or some neighbors :wink: Another thing could be some random channel auto reconfiguration. Will might check some repeater / router logs according to this.

@jsimmonds Do you know if a total loss of the Wi-Fi signal would be logged somewhere?

1 Like

Don’t know but will dig around . . wpa_supplicant on the FBOS Linux is managed by Elixir app :vintage_net_wifi and relations. There should be logging that we could see or enable.

Just thinking bigger-picture about the :nxdomain abort from HTTP Client here

  • Should DNS client query requests normally be re-tried n-times on response timeout ?
  • Are there other signs in latest Wireshark capture that indicate a loss of internet apart from 18.205.222.128 unexpectedly REJecting ?

What I’m wondering is whether AWS load-balancing is working correctly ?

Edit: My mistake there: If NO Internet then AWS cannot REJect :blush: