How To Troubleshoot AutoID

Author:

Hope Sun

Created at:

Aug 2023

Updated at:

Jan 2024

Overview

Whether you’re an experienced professional or just starting out, this video tutorial will guide you through a range of troubleshooting tips and techniques for AutoID. These techniques help you learn to identify problems, pinpoint root causes, and implement troubleshooting solutions using practical real-world examples.

AutoID Troubleshooting Checklist:

The AutoID Troubleshooting Checklist below serves as a reference, reflecting the steps outlined in the video.

Note: These steps assume a Unix-like system with required permissions and configurations.

  1. Check Certificates

    • Set the export password: substitute the example “password” with your actual password value:

      export password=password

    • Navigate to the MongoDB certificates directory:

      cd mongo/certs

    • Verify the MongoDB certificate against the CA file:

      openssl verify -CAfile ca.pem mongonodekey.crt_signed

    • List contents of MongoDB client keystore:

      keytool -list -keystore mongo-client-keystore.jks -storepass $password

    • List contents of MongoDB server truststore:

      keytool -list -keystore mongo-server-truststore.jks -storepass $password

    • Navigate to the OpenSearch certificates directory:

      cd opensearch/certs

    • List contents of OpenSearch client keystore:

      keytool -list -keystore elastic-client-keystore.jks -storepass $password

    • List contents of OpenSearch server truststore:

      keytool -list -keystore elastic-server-truststore.jks -storepass $password

    • Verify the OpenSearch certificate against the CA file:

      openssl verify -CAfile root-ca.pem esnodekey.crt_signed

  2. Verify Spark and OpenSearch Master

    • Check if Spark is running on port 7077:

      netstat -tunlp

    • Access Spark master web UI to verify status: (verify if the master is up and a worker joined)

      elinks http://localhost:8080

  3. Check Livy

    • Check Livy functionality by creating a PySpark session:

      • Request session creation:

        curl -X POST --data '{"kind": "pyspark"}' -H "Content-Type: application/json" 10.138.0.65:8998/sessions

      • Verify session creation response.

    • Execute a PySpark statement in the session:

      • Submit statement:

        curl 10.138.0.65:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"1 + 2"}'

      • Verify statement execution response.

    • Check statement output:

      • Get statement result:

        curl 10.138.0.65:8998/sessions/0/statements/0 | python3.8 -m json.tool

      • Ensure output data contains the value 3; livy successfully connected to spark.

  4. Check Cassandra

    • Check Cassandra cluster status using SSL:

      • cluster status:

        cqlsh --ssl 10.138.0.65 -u cassandra -p cassandra -e 'describe cluster'

      • Connect as zoran_user to Cassandra database:

        cqlsh --ssl 10.138.0.65 -u zoran_dba -p 'Welcome!23'

      • Clear system data:

        sudo rm -rf /var/lib/cassandra/data/system/*

  5. Check Mongo Login

    • Access the MongoDB shell using ‘mongoadmin’ credentials:

      mongo -u 'mongoadmin' -p 'Welcome!23' -host 10.138.0.65 --tls --tlsCertificateKeyFile mongodb.pem --tlsCAFile rootCA.pem

      • Make sure you are able to access the Mongo shell.

  6. Check OpenSearch

    • Query OpenSearch endpoint using 'elasticadmin ’ user credentials to ensure you can retrieve information about the indices (databases) stored in the OpenSearch instance:

      curl -k --location --user elasticadmin --request GET https://10.138.0.65:9200/_cat/indices

    • Check the status of the OpenSearch instance with 'elasticadmin ’ user credentials:

      curl -XGET https://10.138.0.65:9200 -u 'elasticadmin:Welcome!23' --cacert root-ca.pem

  7. Check OpenSearch Security Configuration:

    • Navigate to OpenSearch security configuration directory:

      cd opensearch/plugins/opensearch-security/securityconfig

    • Run OpenSearch security update:

      RUN opensearch security update

    • Navigate to OpenSearch security tools directory:

      cd opensearch/plugins/opensearch-security/tools/

    • Update OpenSearch Security Configuration:

      ./securityadmin.sh -cd ../securityconfig/ -icl -nhnv -cacert ../../../config/root-ca.pem -cert ../../../config/admin.pem -key ../../../config/admin-key.pem -h 10.138.0.65

  8. Run Deployer and Restart Services

    • Run deployer script:

      ./run_deployer.sh

      Note: Ensure to remove localhost entry from /etc/hosts.

    • Navigate to Spark home directory:

      cd spark home

    • Stop Spark and Livy services:

      ./spark/sbin/stop-all.sh
./livy/bin/livy-server stop

    • Start Spark and Livy services:

      ./spark/sbin/start-all.sh
./livy/bin/livy-server start

    • Start MongoDB:

      mongod --config /mongo/mongo.conf

    • Start OpenSearch:

      ./opensearch/opensearch-tar-install.sh > opensearch.log 2>&1 &

    • Navigate to OpenSearch Dashboards bin directory:

      cd ~/opensearch-dashboards/bin/

    • Start OpenSearch Dashboards:

      ./opensearch-dashboards --host 10.138.0.65 > opensearch-dashboards.log 2>&1 &
      +

Additional resources

image Autonomous Identity

Community

Documentation

Getting Started

Deploy and Install

Configure and Maintain

Using Autonomous Identity

Reference