MQTT & Palo Alto NGFW

it.general.smc · February 11, 2025, 6:21pm

Hi All,

New to FarmBot here.

I am the SysAdmin for an Australian College. We have a farm bot connected via Ethernet to our Enterprise Network, which routes out to the internet via a Palo Alto NGFW.
Our STEM team has mentioned to us that the farm bot has gone offline over Christmas, with the only change been a PAN NGFW OS upgrade.
The farm bot is on its own VLAN and PAN OS allows all outbound traffic at the moment.
The issue is we are seeing alternate session end messages for all the MQTT traffic, alternating between aged-out and tcp-rst-from-client. Accordingly, the f\arm bot cloud console (browser) is showing alternating disconnected and connected messages.
We cannot control the farm bot.

We don’t feel it’s the PAN-OS version; everything else, over 2000 devices, windows, Linux, apple, etc. are running just fine.

I have tested different farm bot genesis versions, currently running Farmduino 1.7.

Are we able to change a setting to force MQTT over a non SSL port of 1883? or is MQTT over SSL 8883 mandatory?

Is there anything else we can try to resolve this issue with out contacting PAN Support or changing PAN OS for 1 device.

Thank you for your time.

jsimmonds · February 12, 2025, 4:16am

Hi @it.general.smc

FarmBot people need to respond to this but seems to me you might need to be able to build your own FarmbotOS and to configure it the way you need.
The JWT token supplied by FarmBot API defines the MQTT host and the code hardwires SSL and port 8883 if the token includes the mqtt_ws key.

Take a look at the GitHub code which sets up the MQTT client, e.g.

github.com

FarmBot/farmbot_os/blob/b42e9abd5462c4aac2f149b9acd41dbfa3a0e615/lib/ext/mqtt/supervisor.ex#L26


      
            Supervisor.init(children(), strategy: :one_for_all)
          end
          
          def children do
            token = Config.get_config_value(:string, "authorization", "token")
            config = Application.get_env(:farmbot, __MODULE__) || []
          
            Keyword.get(config, :children, [mqtt_child(token)])
          end
          
          def mqtt_child(raw_token) do
            token = JWT.decode!(raw_token)
            host = token.mqtt
            username = token.bot
            # EXPERIMENT: Can't isolate cause of MQTT blinking.
            # I am suspicious that it is caused by non-deterministic
            # mqtt client_id. Temporarily making client_id generation
            # deterministic. RC 1 DEC 21.
            # jitter = String.slice(UUID.uuid4(:hex), 0..7)
            # _#{jitter}"
            client_id = "#{token.bot}_#{Project.version()}"

Can you post a tcpdump or Wireshark decode packet summary of those streams that fail ?

Edit

Maybe ignore the above . . other Educational institutions seem to sort out this type of issue with help from the IT Professionals that manage their network . . I’m just a guy who likes to find root causes

it.general.smc · February 16, 2025, 11:24pm

I am one of those IT Professionals who manages the network; SysAdmin = ICT Systems Administrator. It turns out that due to a recent CVE in PAN-OS, we updated our firewall, but it still does not work. 8883 is going out fine or received sporadically.

jsimmonds · February 17, 2025, 3:44am

Oh, Ok
Lets get problem solving

Some ad-hoc questions for you -

Have you asked Palo Alto Customer Support whether this latest update to PAN-OS might have broken something ?
Checked Support Forums for the same issue ?
Coincidences occur . . is it possible, across the weekend, that some new device was added to the network segment that the Farmbot connects on ? (i.e. duplicate MAC or IPv4 addresses ? How locked-down is that segment ?)
Was there a latent firewall configuration change that came into effect after the PAN-OS reboot ?

That’s very non-technical
Have you done a packet capture over several “connection resets” on that “Farmbot” segment ?
That’s the only way we’re going to be able to identify why the MQTT conversations are unreliable.

it.general.smc · February 18, 2025, 4:07am

Thank you for replying.
As stated above in my opening, I have not contacted Palo. It’s one device on the firewall of over 2500 devices. Seems very strange for only 1 device to not work, We thought we would reach out here first.
Yes, I have checked out forums, which led me to this one to seek guidance.
As also stated in the opening. It is on its own network/VLAN. It is the only device in that zone and therefore records show no clashes with anything else (Unless I put my device in that VLAN for packet capture, that would make two, not including the firewall interface of course). It is using a new rule on the firewall, created recently after discovering this problem. That rule is open all out, no decryption, no applications, or profiles assigned. Effectively open access out.
Wireshark shows what Palo is seeing. It is attempting outbound traffic to IP 34.235.99.81, ports. 443, 8883. My internal IP is also hitting that IP and Port on the same FW rule.
A new Rule was created at the top to allow FarmBot out to the world. No other rule is in effect for the FarmBot to the Internet.

I will review Wireshark in more detail and get back to this post in due course.

By the sounds of it though the problem is on our end somewhere.

Thanks.

jsimmonds · February 18, 2025, 5:35am

These just jumped out at me.

Ask the STEM team to make sure that the Farmbot is running the latest version of FarmBot OS for the Farmbot model version that the College wants connected. (e.g. if the device is a Genesis v1.6 hardware kit, then it must run the latest FBOS version [v15.4.10] for the v1.6 kit)

There is a FarmBot OS download ‘wizard’ at https://my.farm.bot/os

it.general.smc · February 18, 2025, 11:18pm

I already took out the card and put that particular OS on it. v1.6 and set it up again, tested both Ethernet and Wi-Fi.

jsimmonds · February 19, 2025, 3:19am

Ok, good.

Is the bot model Express or Genesis ? (each has a different Raspberry Pi model)
Tried running the bot over a network other than the College Ethernet ?
(a Wi-Fi network such as a Telstra mobile phone hotspot should work)
In the Ethernet packet captures, does the bot close (FIN) the MQTT stream ?
Do the packet captures (or the lights inside the Farmbot Electronics box) suggest that the bot is periodically rebooting ?

Have you read this ?
For IT Security Professionals | FarmBot Software Documentation

system · March 5, 2025, 3:19am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.