Troubleshooting tips

Below is a list of troubleshooting actions you can perform to troubleshoot issues/failures.

Additionally, we would really appreciate if you create a issue.

Please include following information when creating the issue:

• environment yaml definition
• application yaml definition
• cld commands used
• relevant stack trace from cld.log

Troubleshooting actions (in no particular order)

• Restart CaaStle server:

  • cd <cloudark-dir>; ./restart-cloudark.sh

• Check relevant files and directories for errors:

  • install.log: This file contains CaaStle client and server installation logs
  • cld.log: This file contains the server’s runtime logs
  • cloudark-virtenv: This is the Python virtual environment in which CaaStle client and server are installed
  • ~/.cld/data/deployments/cld.sqlite: SQLite database file in which CaaStle’s internal state is stored

• Check deployment related files that are generated by CaaStle

  • CaaStle stores generated files in individual directories inside the ‘~/.cld/data/deployments’ folder

• Check if all the CaaStle’s cloud extensions have loaded correctly

  $ source cloudark-virtenv/bin/activate

  $ epi group list

  You should see list of all the loaded stevedore extensions. CaaStle extensions are available under “server.server_plugins” namespace. Some example commands to check extensions are below.

  $ epi group show server.server_plugins.aws.coe

  Shows the aws.coe extensions

  $ epi group show server.server_plugins.gcloud.app

  Shows the gcloud.app extensions

• Use environment-specific shell to run the target cloud’s native CLI commands

• Install and start CaaStle in fresh virtual environment manually:

  $ virtualenv freshinstall

  $ source freshinstall/bin/activate

  $ pip install -r requirements.txt

  $ python client/setup.py install

  $ python server/setup.py install

  $ python server/fmserver.py

  Open another shell window, activate the freshinstall virtual environment, and then run client commands. Note that you need to be in the directory where Dockerfile is present to run client commands.

• AWS deployments

  1. Logging to ECS cluster instance:

     As part of ECS cluster creation CaaStle creates a ssh key-pair. You can use that to login to the ECS cluster instance as follows:

     • Find out location of the .pem file for your cluster - cld environment show <env-name>
     • cd into the directory where the .pem file is located
     • Change permissions of the .pem file - chmod 400 <pem file>
     • Find out IP of the cluster instance (available from cld environment show command)
     • SSH into the cluster instance - ssh -i “<pem file>” ec2-user@<AWS cluster instance IP>

  2. app_url vs. app_ip_url:

     CaaStle creates an application load balancer (alb) for each deployment. app_url is formed with this alb. app_ip_url is constructed using the IP address of the cluster node on which the application is deployed. We have observed that the alb based app_url takes some time to start responding. The app_ip_url can be used till then.

• Debug flag for CLI commands

  You can include ‘debug’ flag with any CaaStle command to see the complete debug trace of its execution. Example:

  $ cld env create env-aws environment-ecs.yaml --debug
  

• Single stepping

  Use these steps to single step through the code:

  • Stop CaaStle server: ./stop-cloudark.sh
  • Open the file in which you want to set a breakpoint
  • Set the breakpoint at appropriate line: import pdb; pdb.set_trace()
  • Start the server directly: python server/fmserver.py
  • If you have put breakpoint in client-side code then you will have to install the client in the virtual environment again: python client/setup.py install
  • Execute required client command