Problems when importing sequences via REST Api


#1

Hi,

I am running the Farmbot software on a local deployment and want to migrate it to another installation. To this end, I exported all relevant data via the export function described here: Copy data from your webserver to mine.

Since there is no import function, I wrote a simple script which imports all the data stored in the export json-file via the REST API to my new installation of the Farmbot software. Everything looks fine so far and I get status OK for all my import requests.

However, when I try to connect the Farmbot OS (I mean the Raspi) with this installation I have problems when syncing the sequences. I have approx. 700 sequences imported to the new installation. When I delete some of these sequences the syncing worked. It worked pretty stable with approx. 400 sequences. Sometimes the syncing even works with 600 sequences but never with all my 700 sequences.

So my questions are:

  • Is the syncing issue somehow related to an issue with the data import? Or is it a sperate problem?
  • As far as I understand, the syncing process it works as follows: the browser app fetches all resources via the REST API and puts the information to the message queue from which the Raspi is collecting it. Is this correct?
  • How can I troubleshoot the syncing issue?

Thanks for your help!


#2

@wookie182

Is the syncing issue somehow related to an issue with the data import? Or is it a sperate problem?

I don’t have enough information to say, but it is possible. Do you have any error logs from your server, your browser or FBOS?

As far as I understand, the syncing process it works as follows: the browser app fetches all resources via the REST API and puts the information to the message queue from which the Raspi is collecting it. Is this correct?

Data is stored in the REST API. FBOS and the browser fetch data from the API independantly. If the browser updates or creates data, the change is broadcasted by the API to all message broker subscribers so that they can have fresh data. This requires auto sync to be enabled on FBOS.

How can I troubleshoot the syncing issue?

Take a look at your server logs when the sync is taking place. It would be a good idea to clear your log buffer before starting the sync, since it will be a fairly large log. Do you see any “Record not found” errors? My guess is that one of your sequences references a non-existent foreign key from the old server. Example: a CeleryScript node with a tool_id of 5, even though there is no such tool on the new server (or if there is, it is not accesible by the device because it is not the resource owner).

How does your importer script handle translation of tool_id, point_id, sequence_id etc?


#3

Thanks @RickCarlino !

My script does not support translation of tool_id, etc. My strategy is to start from a clean database and sort the commited datasets by id, which means that the id of one dataset is the same as in the old database (if there are gaps, I create the dataset and delete it afterwards). I have observed, if there are inconsistencies between the ids it is not possible to commit such an inconsistant dataset successfully. I always recieved status 422 instead. Since I get status OK for all my committed datasets I dont believe this is an issue. However, I have to admit that I just found a bug when committing the points resource. Maybe this is the issue. I will check somewhen next week and let you know.

I started to write my import script with translation of the keys. But I could not manage it in the end. Where can I find the exact correspondencies between resources and id-references (sequences->sequence_id, tools->tool_id, regimens->regimen_id) to be considered? Is such an import-script still of interest? If it is working in the end I could make it (more) beautiful and publish it.


#4

Hi @RickCarlino

now I can provide more detailed log messages. I am running a dockerized deployment and watch the logs via docker-compose logs …

After pressing the sync button, I get the message “Syncing” and “Starting HTTP requests.” from rabbit_jobs. Then I see requests to the web app to all resources (device, farm_events, sequences, tools, etc.). Then nothing happens and I see “Sync Error” in the browser app. Additionally I see from delayed_job service two “Job log.deliver” messages. But from my point of view this does not seem to be an issue.

When the sync is successfull (which happens when I commit fewer sequences like 200 out of 700) then the only difference is that I get the message “Synced” from the rabbit_jobs service and the button in the browser app turns green.

Do you have any idea why the import goes wrong? Thanks a lot…


#5

@wookie182 I am not sure what would cause that error, based on the information provided. What version of the app are you running?

Also, to answer your question on syncing resources, here is the load order we use in the Web App to make sure there are no missing foreign keys:

  1. User
  2. Device
  3. FbosConfig
  4. FirmwareConfig
  5. FarmwareEnv
  6. FarmwareInstallation
  7. WebAppConfig
  8. SavedGarden
  9. PlantTemplate
  10. Peripheral
  11. Point
  12. Sensor
  13. Tool
  14. Alert
  15. SensorReading
  16. Sequence :arrow_backward: THIS ONE IS SPECIAL. See notes below.*
  17. Regimen
  18. PinBinding
  19. FarmEvent
  20. Image
  21. Log
  22. WebcamFeed

*If you were to attempt a sequence importer, you would probably need to do something along these steps:

  1. Save each sequence from the old server by name, but leave the body empty. This is to account for the fact that sequences can reference other sequences. They need to have an ID to be referenced.
  2. Recurse (not loop) into the sequence and collect all of the pin_id, pointer_id, sequence_id, tool_id values inside of sequence.body. This will not work without recursion due to the deeply nested structure of sequences. A simple loop is not adequate here. An example of a CeleryScript tree climber can be found here.
  3. Once you have collected all of the pin_id, pointer_id, sequence_id, tool_id values, you can translate the old server’s value to the new server’s value and save the sequence for a second time.

If you have not done so already, please take a look at the Official CeleryScript Documentation. I am happy to answer any questions you have.


#6

Hi @RickCarlino,

thanks for your explanation. As far as I am aware my script works according to your descritption. However, the issue still exists.

My fundamental use-case is to migrate my data from a local deployment to a cloud-based deployment. Now, I tried a data migration not via the REST API but to copy the docker volumens of the database directly from my local installation to the cloud-based installation. This worked initially fine (snycing successful and I was able to execute sequences). But after manually adding and deleting a sequence via the browser app, the sync error occured again. The log messages are pretty much the same as described above.

This seems to me that the issue is rather based on the huge number of sequences (approx. 700) I use or on the connectivity of my cloud-based installtion. However, my local deployment works fine and stable. Do you have any further ideas how to fix this issue?


#7

@wookie182 This sounds plausible. 700 sequences is an extremely high sequence count. I ran a query of all active users in the last 30 days, and a sequence count higher than 170 is almost unheard of. On the flip side of that, I do know of a user that does have a relatively high sequence count (~300) and has mentioned occasional failures while syncing, which is fixed by re-attempting the sync. Obviously, This is less than ideal for users with high sequence counts and we are working on a long term solution in the next release of FarmBot v8.

The long term solution is the “variables” feature, where instead of making a sequence for every plant in your garden, you will create a single generic sequence (“Water a Plant”, “Pickup a Tool”, etc) and then pass each plant/point/tool in as a variable. This will cut down on the total number of sequences required substantially.

I can’t give an estimate on a release date, but that feature is our only priority right now and it will be available with the release of FarmBot OS v8.

If a high sequence count is indeed the cuase of the issue, your options would be to:

  • Minimize your sequence count or wait for the v8 release.
  • Build a custom Farmware that can accomplish the task for your setup with a smaller number of sequences.

There are also third party Farmwares available to help with looping over large sets of plants, such as MLH and Loop-Plant-With-Filters but these are not officially supported by FarmBot, Inc.

Update: I spoke with @connor this morning about the issue and he mentioned that FBOS v8 has new systems in place to deal with higher sequence counts, in addition to the “variables” feature mentioned above.