I just ran a long sequence taking pictures of the bed.
Today, i still loose some ping, bu not always every 18s (see trace).
But a MQTT ping over TCP should succeed even when loosing a packet every 18s, TCP has flow control and retry to deal with that.
When taking pictures, pictures are available before the bot moves to the position where the picture was taken meaning that uploading a picture is faster than sending the bot position through MQTT, probably because Google Cloud Storage is faster than your MQTT heroku gateway (i mean the server or the network).
Moreover, the bot seems to have connexion issue until all accumulated bot positions are not sent to the MQTT gateway, and when the bot queue is full or plant status update timeout, the sequence fails. This is a basic flow control issue in your design, there is no back pressure to slow down the sequence until the sequence fails. Of course, this does not happens in your labs with a good latency to your MQTT gateway, but i’m in France, and of course, and internet latency is far longer than in US.
There is only 2 ways to design it properly.
A - the sequence can work off line, and asynchronously upload positions and other data. That means buffering data in the farmbot.
B - the sequence works online, and wait data to be uploaded to do the next step.
It seems you designed something with a limited queue of data to upload on the bot, and no back pressure to slow down the running sequence. As the result, if a sequence produce more data than the network can upload, after a while, the queue is full and the sequence fails. Of course, in your labs, this phenomena does not happen, but from France, it happens, and this is very frustrating to get a farmbot that does not work because such of basic design flaw. Please consider this explanation to improve your bot, with a customisable MQTT timeout, and introduce a new sequence command like “wait until queue is empty” that could be used to slow down the sequence execution in such situation…
64 bytes from 192.168.1.32: icmp_seq=224 ttl=64 time=2.635 ms
64 bytes from 192.168.1.32: icmp_seq=225 ttl=64 time=42.232 ms
64 bytes from 192.168.1.32: icmp_seq=226 ttl=64 time=46.045 ms
64 bytes from 192.168.1.32: icmp_seq=227 ttl=64 time=3.913 ms
64 bytes from 192.168.1.32: icmp_seq=228 ttl=64 time=2.981 ms
64 bytes from 192.168.1.32: icmp_seq=229 ttl=64 time=2.476 ms
64 bytes from 192.168.1.32: icmp_seq=230 ttl=64 time=6.238 ms
64 bytes from 192.168.1.32: icmp_seq=231 ttl=64 time=5.566 ms
64 bytes from 192.168.1.32: icmp_seq=232 ttl=64 time=86.654 ms
64 bytes from 192.168.1.32: icmp_seq=233 ttl=64 time=2.324 ms
64 bytes from 192.168.1.32: icmp_seq=234 ttl=64 time=5.772 ms
Request timeout for icmp_seq 235
64 bytes from 192.168.1.32: icmp_seq=236 ttl=64 time=8.503 ms
64 bytes from 192.168.1.32: icmp_seq=237 ttl=64 time=6.262 ms
64 bytes from 192.168.1.32: icmp_seq=238 ttl=64 time=43.793 ms
64 bytes from 192.168.1.32: icmp_seq=239 ttl=64 time=23.265 ms
64 bytes from 192.168.1.32: icmp_seq=240 ttl=64 time=315.803 ms
64 bytes from 192.168.1.32: icmp_seq=241 ttl=64 time=3.028 ms
64 bytes from 192.168.1.32: icmp_seq=242 ttl=64 time=4.383 ms
64 bytes from 192.168.1.32: icmp_seq=243 ttl=64 time=2.328 ms
64 bytes from 192.168.1.32: icmp_seq=244 ttl=64 time=2.938 ms
64 bytes from 192.168.1.32: icmp_seq=245 ttl=64 time=6.837 ms
64 bytes from 192.168.1.32: icmp_seq=246 ttl=64 time=106.957 ms
64 bytes from 192.168.1.32: icmp_seq=247 ttl=64 time=3.792 ms
Request timeout for icmp_seq 248
64 bytes from 192.168.1.32: icmp_seq=249 ttl=64 time=63.010 ms
64 bytes from 192.168.1.32: icmp_seq=250 ttl=64 time=255.016 ms
64 bytes from 192.168.1.32: icmp_seq=251 ttl=64 time=339.240 ms
64 bytes from 192.168.1.32: icmp_seq=252 ttl=64 time=3.556 ms
64 bytes from 192.168.1.32: icmp_seq=253 ttl=64 time=2.554 ms
64 bytes from 192.168.1.32: icmp_seq=254 ttl=64 time=2.520 ms
64 bytes from 192.168.1.32: icmp_seq=255 ttl=64 time=151.783 ms
64 bytes from 192.168.1.32: icmp_seq=256 ttl=64 time=687.512 ms
64 bytes from 192.168.1.32: icmp_seq=257 ttl=64 time=31.304 ms
64 bytes from 192.168.1.32: icmp_seq=258 ttl=64 time=16.051 ms
64 bytes from 192.168.1.32: icmp_seq=259 ttl=64 time=9.169 ms
Request timeout for icmp_seq 260
64 bytes from 192.168.1.32: icmp_seq=261 ttl=64 time=27.119 ms
64 bytes from 192.168.1.32: icmp_seq=262 ttl=64 time=19.159 ms
64 bytes from 192.168.1.32: icmp_seq=263 ttl=64 time=6.111 ms
64 bytes from 192.168.1.32: icmp_seq=264 ttl=64 time=9.765 ms
64 bytes from 192.168.1.32: icmp_seq=265 ttl=64 time=926.713 ms
64 bytes from 192.168.1.32: icmp_seq=266 ttl=64 time=9.777 ms
64 bytes from 192.168.1.32: icmp_seq=267 ttl=64 time=6.542 ms
64 bytes from 192.168.1.32: icmp_seq=268 ttl=64 time=4.574 ms
64 bytes from 192.168.1.32: icmp_seq=269 ttl=64 time=3.793 ms
64 bytes from 192.168.1.32: icmp_seq=270 ttl=64 time=14.032 ms
64 bytes from 192.168.1.32: icmp_seq=271 ttl=64 time=3.203 ms
64 bytes from 192.168.1.32: icmp_seq=272 ttl=64 time=3.407 ms
64 bytes from 192.168.1.32: icmp_seq=273 ttl=64 time=3.169 ms
64 bytes from 192.168.1.32: icmp_seq=274 ttl=64 time=3.098 ms
64 bytes from 192.168.1.32: icmp_seq=275 ttl=64 time=5.607 ms
Request timeout for icmp_seq 276
64 bytes from 192.168.1.32: icmp_seq=277 ttl=64 time=7.401 ms
64 bytes from 192.168.1.32: icmp_seq=278 ttl=64 time=11.029 ms
64 bytes from 192.168.1.32: icmp_seq=279 ttl=64 time=4.222 ms
64 bytes from 192.168.1.32: icmp_seq=280 ttl=64 time=10.268 ms
64 bytes from 192.168.1.32: icmp_seq=281 ttl=64 time=17.182 ms
64 bytes from 192.168.1.32: icmp_seq=282 ttl=64 time=9.815 ms
64 bytes from 192.168.1.32: icmp_seq=283 ttl=64 time=9.102 ms
64 bytes from 192.168.1.32: icmp_seq=284 ttl=64 time=165.902 ms
64 bytes from 192.168.1.32: icmp_seq=285 ttl=64 time=3.680 ms
64 bytes from 192.168.1.32: icmp_seq=286 ttl=64 time=12.883 ms
64 bytes from 192.168.1.32: icmp_seq=287 ttl=64 time=7.640 ms
64 bytes from 192.168.1.32: icmp_seq=288 ttl=64 time=11.075 ms
64 bytes from 192.168.1.32: icmp_seq=289 ttl=64 time=3.650 ms
64 bytes from 192.168.1.32: icmp_seq=290 ttl=64 time=9.800 ms
64 bytes from 192.168.1.32: icmp_seq=291 ttl=64 time=6.388 ms
64 bytes from 192.168.1.32: icmp_seq=292 ttl=64 time=9.398 ms
64 bytes from 192.168.1.32: icmp_seq=293 ttl=64 time=15.544 ms
64 bytes from 192.168.1.32: icmp_seq=294 ttl=64 time=10.502 ms