How To: Troubleshoot AutoID

Overview

This comprehensive video tutorial provides essential techniques for troubleshooting your Autonomous Identity (AutoID) deployment effectively.

Whether you’re an experienced professional or just starting out, this video guides you through a range of troubleshooting tips and techniques. These techniques help you learn to identify problems, pinpoint root causes, and implement solutions using practical real-world examples. By watching this demonstration, you’ll gain actionable skills that enable you to confidently apply troubleshooting techniques to your AutoID deployment.

AutoID Troubleshooting Checklist:

The AutoID Troubleshooting Checklist below serves as a reference, reflecting the steps outlined in the video.

Note: These steps assume a Unix-like system with required permissions and configurations.

  1. Check Certificates

    • Set the export password: substitute the example “password” with your actual password value:

      export password=password
      
    • Navigate to the MongoDB certificates directory:

      cd mongo/certs
      
    • Verify the MongoDB certificate against the CA file:

      openssl verify -CAfile ca.pem mongonodekey.crt_signed
      
    • List contents of MongoDB client keystore:

      keytool -list -keystore mongo-client-keystore.jks -storepass $password
      
    • List contents of MongoDB server truststore:

      keytool -list -keystore mongo-server-truststore.jks -storepass $password
      
    • Navigate to the OpenSearch certificates directory:

      cd opensearch/certs
      
    • List contents of OpenSearch client keystore:

      keytool -list -keystore elastic-client-keystore.jks -storepass $password
      
    • List contents of OpenSearch server truststore:

      keytool -list -keystore elastic-server-truststore.jks -storepass $password
      
    • Verify the OpenSearch certificate against the CA file:

      openssl verify -CAfile root-ca.pem esnodekey.crt_signed
      
  2. Verify Spark and OpenSearch Master

    • Check if Spark is running on port 7077:

      netstat -tunlp
      
    • Access Spark master web UI to verify status: (verify if the master is up and a worker joined)

      elinks http://localhost:8080
      
  3. Check Livy

    • Check Livy functionality by creating a PySpark session:

      • Request session creation:
        curl -X POST --data '{"kind": "pyspark"}' -H "Content-Type: application/json" 10.138.0.65:8998/sessions
        
      • Verify session creation response.
    • Execute a PySpark statement in the session:

      • Submit statement:
        curl 10.138.0.65:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"1 + 2"}'
        
      • Verify statement execution response.
    • Check statement output:

      • Get statement result:
        curl 10.138.0.65:8998/sessions/0/statements/0 | python3.8 -m json.tool
        
      • Ensure output data contains the value 3; livy successfully connected to spark.
  4. Check Cassandra

    • Check Cassandra cluster status using SSL:
      • cluster status:
        cqlsh --ssl 10.138.0.65 -u cassandra -p cassandra -e 'describe cluster'
        
      • Connect as zoran_user to Cassandra database:
        cqlsh --ssl 10.138.0.65 -u zoran_dba -p 'Welcome!23'
        
      • Clear system data:
        sudo rm -rf /var/lib/cassandra/data/system/*
        
  5. Check Mongo Login

    • Access the MongoDB shell using ‘mongoadmin’ credentials:
      mongo -u 'mongoadmin' -p 'Welcome!23' -host 10.138.0.65 --tls --tlsCertificateKeyFile mongodb.pem --tlsCAFile rootCA.pem
      
      • Make sure you are able to access the Mongo shell.
  6. Check OpenSearch

    • Query OpenSearch endpoint using 'elasticadmin ’ user credentials to ensure you can retrieve information about the indices (databases) stored in the OpenSearch instance:

      curl -k --location --user elasticadmin --request GET https://10.138.0.65:9200/_cat/indices
      
    • Check the status of the OpenSearch instance with 'elasticadmin ’ user credentials:

      curl -XGET https://10.138.0.65:9200 -u 'elasticadmin:Welcome!23' --cacert root-ca.pem
      
  7. Check OpenSearch Security Configuration:

    • Navigate to OpenSearch security configuration directory:

      cd opensearch/plugins/opensearch-security/securityconfig
      
    • Run OpenSearch security update:

      RUN opensearch security update
      
    • Navigate to OpenSearch security tools directory:

      cd opensearch/plugins/opensearch-security/tools/
      
    • Update OpenSearch Security Configuration:

      ./securityadmin.sh -cd ../securityconfig/ -icl -nhnv -cacert ../../../config/root-ca.pem -cert ../../../config/admin.pem -key ../../../config/admin-key.pem -h 10.138.0.65
      
  8. Run Deployer and Restart Services

    • Run deployer script:
      ./run_deployer.sh
      

    Note: Ensure to remove localhost entry from /etc/hosts.

    • Navigate to Spark home directory:

      cd spark home
      
    • Stop Spark and Livy services:

      ./spark/sbin/stop-all.sh
      ./livy/bin/livy-server stop
      
    • Start Spark and Livy services:

      ./spark/sbin/start-all.sh
      ./livy/bin/livy-server start
      
    • Start MongoDB:

      mongod --config /mongo/mongo.conf
      
    • Start OpenSearch:

      ./opensearch/opensearch-tar-install.sh > opensearch.log 2>&1 &
      
    • Navigate to OpenSearch Dashboards bin directory:

      cd ~/opensearch-dashboards/bin/
      
    • Start OpenSearch Dashboards:

      ./opensearch-dashboards --host 10.138.0.65 > opensearch-dashboards.log 2>&1 &
      

Additional resources

image Autonomous Identity

Community

Documentation

Getting Started
Deploy and Install
Configure and Maintain
Using Autonomous Identity
Reference
2 Likes

Great resource, thanks Hope!

1 Like