Overview
Whether you’re an experienced professional or just starting out, this video tutorial will guide you through a range of troubleshooting tips and techniques for AutoID. These techniques help you learn to identify problems, pinpoint root causes, and implement troubleshooting solutions using practical real-world examples.
AutoID Troubleshooting Checklist:
The AutoID Troubleshooting Checklist below serves as a reference, reflecting the steps outlined in the video.
Note: These steps assume a Unix-like system with required permissions and configurations.
-
Check Certificates
-
Set the export password: substitute the example “password” with your actual password value:
export password=password
-
Navigate to the MongoDB certificates directory:
cd mongo/certs
-
Verify the MongoDB certificate against the CA file:
openssl verify -CAfile ca.pem mongonodekey.crt_signed
-
List contents of MongoDB client keystore:
keytool -list -keystore mongo-client-keystore.jks -storepass $password
-
List contents of MongoDB server truststore:
keytool -list -keystore mongo-server-truststore.jks -storepass $password
-
Navigate to the OpenSearch certificates directory:
cd opensearch/certs
-
List contents of OpenSearch client keystore:
keytool -list -keystore elastic-client-keystore.jks -storepass $password
-
List contents of OpenSearch server truststore:
keytool -list -keystore elastic-server-truststore.jks -storepass $password
-
Verify the OpenSearch certificate against the CA file:
openssl verify -CAfile root-ca.pem esnodekey.crt_signed
-
-
Verify Spark and OpenSearch Master
-
Check if Spark is running on port 7077:
netstat -tunlp
-
Access Spark master web UI to verify status: (verify if the master is up and a worker joined)
elinks http://localhost:8080
-
-
Check Livy
-
Check Livy functionality by creating a PySpark session:
- Request session creation:
curl -X POST --data '{"kind": "pyspark"}' -H "Content-Type: application/json" 10.138.0.65:8998/sessions
- Verify session creation response.
- Request session creation:
-
Execute a PySpark statement in the session:
- Submit statement:
curl 10.138.0.65:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"1 + 2"}'
- Verify statement execution response.
- Submit statement:
-
Check statement output:
- Get statement result:
curl 10.138.0.65:8998/sessions/0/statements/0 | python3.8 -m json.tool
- Ensure output data contains the value 3; livy successfully connected to spark.
- Get statement result:
-
-
Check Cassandra
-
Check Cassandra cluster status using SSL:
- cluster status:
cqlsh --ssl 10.138.0.65 -u cassandra -p cassandra -e 'describe cluster'
- Connect as zoran_user to Cassandra database:
cqlsh --ssl 10.138.0.65 -u zoran_dba -p 'Welcome!23'
- Clear system data:
sudo rm -rf /var/lib/cassandra/data/system/*
- cluster status:
-
-
Check Mongo Login
- Access the MongoDB shell using ‘mongoadmin’ credentials:
mongo -u 'mongoadmin' -p 'Welcome!23' -host 10.138.0.65 --tls --tlsCertificateKeyFile mongodb.pem --tlsCAFile rootCA.pem
- Make sure you are able to access the Mongo shell.
- Access the MongoDB shell using ‘mongoadmin’ credentials:
-
Check OpenSearch
-
Query OpenSearch endpoint using 'elasticadmin ’ user credentials to ensure you can retrieve information about the indices (databases) stored in the OpenSearch instance:
curl -k --location --user elasticadmin --request GET https://10.138.0.65:9200/_cat/indices
-
Check the status of the OpenSearch instance with 'elasticadmin ’ user credentials:
curl -XGET https://10.138.0.65:9200 -u 'elasticadmin:Welcome!23' --cacert root-ca.pem
-
-
Check OpenSearch Security Configuration:
-
Navigate to OpenSearch security configuration directory:
cd opensearch/plugins/opensearch-security/securityconfig
-
Run OpenSearch security update:
RUN opensearch security update
-
Navigate to OpenSearch security tools directory:
cd opensearch/plugins/opensearch-security/tools/
-
Update OpenSearch Security Configuration:
./securityadmin.sh -cd ../securityconfig/ -icl -nhnv -cacert ../../../config/root-ca.pem -cert ../../../config/admin.pem -key ../../../config/admin-key.pem -h 10.138.0.65
-
-
Run Deployer and Restart Services
- Run deployer script:
./run_deployer.sh
Note: Ensure to remove localhost entry from /etc/hosts.
-
Navigate to Spark home directory:
cd spark home
-
Stop Spark and Livy services:
./spark/sbin/stop-all.sh ./livy/bin/livy-server stop
-
Start Spark and Livy services:
./spark/sbin/start-all.sh ./livy/bin/livy-server start
-
Start MongoDB:
mongod --config /mongo/mongo.conf
-
Start OpenSearch:
./opensearch/opensearch-tar-install.sh > opensearch.log 2>&1 &
-
Navigate to OpenSearch Dashboards bin directory:
cd ~/opensearch-dashboards/bin/
-
Start OpenSearch Dashboards:
./opensearch-dashboards --host 10.138.0.65 > opensearch-dashboards.log 2>&1 &
- Run deployer script:
Additional resources
Autonomous Identity
Community
Documentation