Hi, to analyze the activities of the bot (how often do we actually perform certain activities) and also for qualified debugging feedback, it would be great to be able to export the logfile (or selected parts of it) as CSV. What do you think?
4 Likes
That’s a great idea @jensGeorgsen. We don’t support CSV export yet, but we do have a JSON API.
Here is how you would download logs in JSON format as they exist today:
NOTE: You must have curl and jq installed for this to work.
# STEP ONE: Extract "token.encoded" from auth token:
TOKEN=`curl -H "Content-Type: application/json" \
-X POST \
-d '{"user":{"email":"jdoe123@gmail.com","password":"123password123"}}' \
https://my.farm.bot/api/tokens | jq ".token.encoded" --raw-output`
# STEP TWO: Download logs from /api/logs endpoint:
curl -H "Authorization: Bearer ${TOKEN}" https://my.farm.bot/api/logs
1 Like
I fear I will have to extend my skill repertoire soon as you mention this API approach often when answering my questions. I understand that you intend to provide CSV export and I believe that this is a good idea as it would make the logs accessible for a broader audience… Thanks for your explanations!
1 Like
@jensGeorgsen Another option (possibly) is to use the “ACCOUNT EXPORT” feature located at the bottom of the account info page:
It’s been a long time since I have used this feature, but I believe logs are included in the data dump (in JSON format).
Please note that the account dump feature does not provide an import feature so it is not to be used as an account backup tool. It may be of interest to you if you want to explore your account data offline, however.
1 Like
And one last thing @jensGeorgsen, since you asked for CSV, it’s worth mentioning that there are many CSV to JSON conversion tools available online. I can’t endorse any one in particular, but they are easily found via search.
Hi @RickCarlino, seems that MS Excel can do that natively. https://youtu.be/CjnsX81qWUg I’ll try that with the file you suggested and report here.
@RickCarlino Thanks for the API code snippet. I got it to work fine with my.farm.bot, but when I use it with my self-hosted WebApp, for the TOKEN it just returns “null”. It appears to connect fine, the port, IP, user, pass is OK. I have API_HOST and API_PORT set, and can reach those ok in a web browser. It just doesn’t return a big token, only the word “null”. Any hints?
I think I know where it comes from. The full return from the API is: {"error":"You can't perform that action. Verify account first"}. So it probably doesn’t allow me to use the API until I verify my email address or something. I just used jebba@localhost on the self-hosted machine, so now I’ll see if I can figure out how to verify that without a mail server… 
I have this set, so it allows me to login fine, but not with the API:
NO_EMAILS=TRUE
Sorry for the noise, but this is how I fixed it, should others encounter this:
# ssh into server
jebba@workstation:~$ ssh farmbot [or whatever your locally hosted WebApp is]
# Connect to Database:
jebba@farmbot:~$ sudo docker exec -it `sudo docker ps | grep postgres | cut -f 1 -d " "` psql -U postgres farmbot_prod
# Update Database to set a time the email was confirmed (make up whatever) for user 1:
farmbot_prod=# UPDATE users SET confirmed_at='2020-05-08 16:12:15.899998' WHERE id=1;
1 Like
@jensGeorgsen Whoops- looks like I had to scale the servers again. This is our highest month of activity ever. Seems to be happening more frequently these days.
@jebba That is on my TODO list for the weeks. Some self-hosters have noted the issues.
Your solution works fine, but a more “Ruby-ish” solution would be to run:
sudo docker-compose run web rails c
and then
User.update_all(confirmed_at: Time.now)
1 Like
Congratulations!
1 Like
It works exactly as described in the video I had linked above.
Just a comment, perhaps an implied suggestion but not a complaint: Seems that the log contains only around 1000 entries. Our garden is currently not completely filled and has more than 340 plants. This means that we will have to limit our analysis to processes that are less than 3 steps per sequence or on parts of the garden.
Thanks anyways! That was a good and quick way to get the data!
2 Likes
I’m glad to hear you got it working!
Yes, this one is unfortunate. In the case of logs, I had to reduce the log count due to some performance issues. The table gets very slow when we set the limit too high. The database is almost certainly missing an index somewhere, but I’ve not had much luck in finding the culprit. As a result, we decided to cap logs at 1,000 for now.
1 Like
Something like this may help to CSV export in the future if needed:
Interesting ! Wonder whether FBOS v10.1.2+ hits the API harder than earlier ( less “featured” ) releases ? I’ll take some measurements. More bots in the field using my.farm.bot also, I suppose.
@jsimmonds That is possible. We’ve been seeing a lt of activity on the /api/points endpoint. My guess is that people are using the new MARK AS feature more now that it can use variables. 