Problem installing LME 2.0

[JGHJME]

I attempted to install LME 2.0 but ran into a problem near the end. This was installed on Ubuntu Server 22.4 with the correct specifications. All went well until I executed this command:

ansible-playbook post_install_local.yml

When I did, I received this error:

TASK [Wait for Fleet API to be ready] ************************************************
fatal: [localhost]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}

PLAY RECAP ***************************************************************************
localhost                  : ok=8    changed=1    unreachable=0    failed=1    skipped=2    rescued=0    ignored=0  

Does anyone know what the problem could be?

[Weastwood]

I had a similar issue and fixed it by removing the inline comments in /etc/containers/systemd/. They install default with an inline comment which causes them to not parse correctly.

Nov 13 19:58:41 - systemd[1]: /run/systemd/generator/lme-kibana.service:21: Failed to parse TimeoutStartSec= parameter, ignoring: 5400 #30 minutes, helps ensure does not fail due to health check

I got rid of #30 minutes, helps ensure does not fail due to health check and that seemed to fix it.

[aarz-snl]

@Weastwood Thanks for pointing that out -- it seems that could in fact be an issue.

@JGHJME You can try running post install again if everything is up. Sometimes this issue is caused by fleet just not being up and ready yet or a hiccup during install. I would certainly try to do a reboot/rerun the post install yaml again. If that still doesn't work You could also try using the uninstall steps then do a reinstall.

Just before you do the reinstall go into /LME/quadlet/lme-kibana.container and remove the comment that says "#30 minutes, helps ensure does not fail due to health check" -- then proceed with install as usual to try it out.

We will do some testing on this. That error may just be cosmetic -- should be fixed but im not entirely sure thats causing the install issue. @Weastwood was your error exactly the same from the ansible playbook?

[aarz-snl]

Before you run post install when you run something like

sudo -i podman ps you want it to look like this:

CONTAINER ID  IMAGE                               COMMAND               CREATED        STATUS                  PORTS                                                                                                                               NAMES
7a239f28b92e  localhost/elasticsearch:LME_LATEST  eswrapper             6 minutes ago  Up 6 minutes (healthy)  0.0.0.0:9200->9200/tcp, 9200/tcp, 9300/tcp                                                                                          lme-elasticsearch
19ca5a08bf26  localhost/elastalert2:LME_LATEST                          6 minutes ago  Up 6 minutes                                                                                                                                                lme-elastalert2
7cf2b73b91df  localhost/wazuh-manager:LME_LATEST                        6 minutes ago  Up 6 minutes (healthy)  0.0.0.0:1514-1515->1514-1515/tcp, 0.0.0.0:55000->55000/tcp, 0.0.0.0:514->514/udp, 514/udp, 1514/tcp, 1515/tcp, 1516/tcp, 55000/tcp  lme-wazuh-manager
6f0ef69b278f  localhost/kibana:LME_LATEST         /usr/local/bin/ki...  6 minutes ago  Up 6 minutes (healthy)  0.0.0.0:443->5601/tcp, 0.0.0.0:5601->5601/tcp, 5601/tcp                                                                             lme-kibana
8c5d970271fa  localhost/elastic-agent:LME_LATEST                        5 minutes ago  Up About a minute       0.0.0.0:8220->8220/tcp                                                                                                              lme-fleet-server

You want kibana/elasticsearch up and healthy, elastic agent should be up for a minute or so. Then run post install yml. Keep in mind just looking at my example I just ran it took upwards of 7 minutes or so before it was ready for post install.

Id recommend waiting until kibana/elasticsearch are up and fleet agent has been running for 4-minutes. This lets all the background processes take place.

[aarz-snl]

Additionally, i recommend moving this problem to an 'issue' instead of discussion

[aarz-snl]

I do recommend removing the inline comments -- and we have an update coming soon with that fix -- but you can do this simply but going into the quadlet files before install and delete the comment mentioned above

[JGHJME]

Today I created a new server and re-pulled an updated LME. All went well until the final script where I revived this error:

TASK [Copy dashboards files /opt/lme/dashboards] ************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: If you are using a module and expect the file to exist on the remote, see the remote_src option
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Could not find or access '~/LME/dashboards/' on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"}

PLAY RECAP **************************************************************************************************************************************************
localhost                  : ok=29   changed=3    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0 

Before ok=8 now ok =29 so it looks like I got a little farther.

[aarz-snl]

Not sure what you mean by an updated LME -- we havent released a fixed version yet.

Do you quadlets still have the inline comments? have you waited at least 10 minutes before running the 2nd ansible playbook? That part of the playbook is just loading the dashboards. You can rerun that playbook at any time and try again.

However, if you're running it during a time that kibana is rebooting / not fully up/ the api upload will fail. Also, it apears to be saying it cant locate ~/LME/dashboards

Does that path exist? It should exist with dashboards included -- if it does it could be some kind of permissions issues

You can add -vvv to your ansible commands for more detail logging and place them in here