# How to Debug and Troubleshoot SocketXP

Here in this section, you'll find all the information you need to debug and troubleshoot any problem with the SocketXP agent.

# SocketXP Config File Location

If you used the single-touch installation script socketxp_install.sh to install and configure SocketXP IoT agent in your IoT or Raspberry Pi device, you can find the configuration file at /etc/socketxp/config.json.

A sample config.json file would look like the following:

$ cat /etc/socketxp/config.json

{
"iot_device_name": "device12345",
"iot_device_group": "temp-sensor",
"tunnels" : [
  {
      "destination": "tcp://127.0.0.1:22"
  }
]
}

The config.json file contents are in JSON file format. More information about the contents of the config.json file can be found here.

Note:

iot_device_name and iot_device_group fields can be set once during the initial device setup. Thereafter if you want to change it, you cannot do so from the config.json file. You need to edit the device in the portal page and update its name, group, customer name and location information and save it.

# SocketXP Device Key File Location:

SocketXP installs a per device private key file at: /var/lib/socketxp/device.key when you login to the SocketXP Cloud Gateway using the SocketXP login command. This key is used to authenticate the device with the user account. The key is also used to uniquely identify the device and the services running in the device that needs remote access.

If you plan to clone your IoT device OS image into multiple SSD or flash cards, make sure you delete this device.key file before cloning.

If you delete a device from the Devices section in the SocketXP Portal page, you also need to delete the /var/lib/socketxp/device.key file in the device before you could add the device back to your account. Otherwise you'll get the following error on your console:

Server Error:  Invalid device key (REclP7PzXMTe61zh3ASa0jd7Y0k3AiF...). Delete device key file at /var/lib/socketxp/device.key and then login again.

# SocketXP Log File

SocketXP IoT agent writes debug logs to a file located at /var/lib/socketxp/socketxp.log

SocketXP agent running in your device connects to the SocketXP IoT Cloud Gateway in the internet via your NAT router or firwall.

On successful connection, you'll see the following debug log in the socketxp.log file:

$ tail -f /var/lib/socketxp/socketxp.log

time="2021-04-09 16:30:10" level=debug msg="Starting SocketXP Client... Connecting local port 22 with SocketXP Cloud Gateway. hostname localhost https false\n\n"

# SocketXP Systemd Logs

SocketXP single-touch installation script socketxp_install.sh installs and runs the SocketXP agent process as a Linux Systemd daemon in the background.

SocketXP agent when run as a background process (as a Linux Systemd Daemon) will write to stdout and stderr which are stored as Systemd logs.

Check the SocketXP agent Systemd status and logs using the following command:

$ systemctl status socketxp

● socketxp.service - SocketXP secure reverse proxy tunnels and webhook proxy service
     Loaded: loaded (/etc/systemd/system/socketxp.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2021-04-09 11:30:10 UTC; 12s ago
   Main PID: 85436 (socketxp)
      Tasks: 6 (limit: 9544)
     Memory: 4.4M
     CGroup: /system.slice/socketxp.service
             └─85436 /usr/local/bin/socketxp --config /root/config.json

Apr 09 11:30:10 socketxp systemd[1]: Started SocketXP secure reverse proxy tunnels and webhook proxy service.
Apr 09 11:30:10 socketxp socketxp[85436]: Using config file: /root/config.json
Apr 09 11:30:10 socketxp socketxp[85436]: Login Succeeded.
Apr 09 11:30:10 socketxp socketxp[85436]: User [] Email [[email protected]].
Apr 09 11:30:10 socketxp socketxp[85436]: Connected.
Apr 09 11:30:10 socketxp socketxp[85436]: TCP tunnel [test-user-sn5siexa] created.
Apr 09 11:30:10 socketxp socketxp[85436]: Access the tunnel using SocketXP agent in IoT Slave Mode.

# SocketXP Syslogs

SocketXP when run as a Systemd service also logs to the /var/log/syslog file. All systemd services running in your device would log to the same syslog file. You could use the below command to grep only the SocketXP syslogs and save it in a separate file.

$ cat /var/log/syslog | grep socketxp > /tmp/socketxp.syslog

A sample SocketXP syslog content might look like the following:

Apr 18 04:30:26 Ubuntu systemd[1]: socketxp.service: Main process exited, code=exited, status=255/n/a
Apr 18 04:30:26 Ubuntu systemd[1]: socketxp.service: Failed with result 'exit-code'.
Apr 18 04:30:31 Ubuntu systemd[1]: socketxp.service: Service hold-off time over, scheduling restart.
Apr 18 04:30:31 Ubuntu systemd[1]: socketxp.service: Scheduled restart job, restart counter is at 29.
Apr 18 04:30:31 Ubuntu socketxp[16506]: Using config file: /home/test-user/config.json
Apr 18 04:30:31 Ubuntu socketxp[16506]: Login Succeeded.
Apr 18 04:30:31 Ubuntu socketxp[16506]: User [] Email [[email protected]].
Apr 18 04:30:31 Ubuntu socketxp[16506]: Could not create service endpoint in SocketXP Cloud Service. Server Error: Device ID already exists; please enter a unique value for device ID.
Apr 18 04:30:31 Ubuntu systemd[1]: socketxp.service: Main process exited, code=exited, status=255/n/a
Apr 18 04:30:31 Ubuntu systemd[1]: socketxp.service: Failed with result 'exit-code'.
Apr 18 04:30:36 Ubuntu systemd[1]: socketxp.service: Service hold-off time over, scheduling restart.
Apr 18 04:30:36 Ubuntu systemd[1]: socketxp.service: Scheduled restart job, restart counter is at 30.
Apr 18 04:30:36 Ubuntu socketxp[16578]: Using config file: /home/test-user/config.json
Apr 18 04:30:36 Ubuntu socketxp[16578]: Login Succeeded.
Apr 18 04:30:36 Ubuntu socketxp[16578]: User [] Email [[email protected]].
Apr 18 04:30:36 Ubuntu socketxp[16578]: Connected.
Apr 18 04:30:36 Ubuntu socketxp[16578]: TCP tunnel [test-user-i3d6l5wn] created.
Apr 18 04:30:36 Ubuntu socketxp[16578]: Access the tunnel using SocketXP agent in IoT Slave Mode.

# SocketXP Linux Systemd Config File

You can find the SocketXP Linux Systemd service configuration file at the following location with the following contents:

$ cat /etc/systemd/system/socketxp.service
[Unit]
Description=SocketXP secure IoT remote access service
Requires=network.target
After=network.target

[Service]
Type=simple
Restart=always
RestartSec=5
ExecStart=/usr/local/bin/socketxp --config /etc/socketxp/config.json
ExecStop=/bin/kill -15 $MAINPID
KillMode=process

[Install]
WantedBy=multi-user.target

# SocketXP Login Error

When you execute the socketxp login command and you get the following error:

$ socketxp login "eyJhbGciOiJIUzI1NiIsInR5c..."
Could not connect to SocketXP Cloud Gateway. Please check internet connectivity or SocketXP Cloud Gateway availability.

There could be multiple reasons for this error.

  1. Your internet connection could be down
  2. Your corporate firewall or proxy server settings is very restrictive and is preventing SocketXP IoT agent (running in your IoT device behind NAT router or Firewall) from reaching the internet. Read the section on Whitelisting SocketXP Service (see below) to make the necessary changes in your firewall or proxy server setttings for SocketXP to work. Most users won't have this problem.
  3. SocketXP IoT Cloud Gateway could be down (which is very very highly unlikely). We strive to keep our service up and runing 100% of the time: 24x7 and 365 days a year. However, SocketXP IoT Cloud Gateway could be down for a brief maintenance. Please note that SocketXP Cloud Gateway will be down for a brief maintenance between the hours of 9:00 PM PST to 12:00 PM PST on Saturdays/Sundays only.

# Whitelisting SocketXP Service

Some corporate networks have a very restrictive firewall or a proxy server that allows outbound traffic to certain domains in the internet and on specific TCP or UDP ports. Outbound traffic from the corporate network to all others domains in the internet, including *.socketxp.com will be restricted. In such situations, you need to request your IT team to add SocketXP service to your corporate firewall's or proxy server's whitelist.

We do not recommend whitelisting our SocketXP IoT Cloud Gateway server's public IP address in your firewall settings. Instead, we recommend you to whitelist our wildcard domain name *.socketxp.com and permit all outbound traffic for TCP ports 80 and 443 in your firewall settings. SocketXP agent communicates with the Cloud Gateway using secure TLS tunnels.

SocketXP Cloud Gateway will never establish any inbound connections to your device behind the firewall. So no modification is required in your inbound firewall rules. SocketXP agent will establish only outbound connections to *.socketxp.com domain in the internet on TCP ports 80 and 443.

# Troubleshooting Legacy SocketXP Agents

In the below sections, you'll find the instructions to troubleshoot SocketXP Agent version 1.3.0 and earlier versions.

# SocketXP Config File Location

If you used the single-touch installation script socketxp_install.sh to install and configure SocketXP IoT agent in your IoT or Raspberry Pi device, you can find the configuration file at /config.json or /root/config.json

A sample legacy config.json file would look like the following:

$ cat $HOME/config.json

{
"authtoken": "<your-auth-token-here>",
"tunnel_enabled": true,
"tunnels" : [
  {
      "destination": "tcp://127.0.0.1:22",
      "protocol": "tcp",
      "custom_domain": "",
      "subdomain": "",
      "iot_device_id": "abc123456789",
      "iot_device_name": "some_unique_device_name",
      "iot_device_group": "some_device_grouping"
  }
],
"relay_enabled": false,
"relays": [{ }]
}

The config.json file contents are in JSON file format. More information about the contents of the config.json file can be found here.

# SocketXP Device Key File Location:

SocketXP installs a per device private key at: $HOME/.socketxp/device.key. This key is used to authenticate the device with the user account. The key is also used to uniquely identify the device and the services running in the device.

If you delete a device from the Devices section in the SocketXP Portal page, you need to delete the $HOME/.socketxp/device.key file also before you add the device back to your account. Otherwise you'll get an error as shown below:

Server Error:  Invalid device key (REclP7PzXMTe61zh3ASa0jd7Y0k3AiF...). Delete device key file at $HOME/.socketxp/device.key and re-login.

Note:

Please confirm your $HOME by executing echo $HOME before you use it. When you run SocketXP agent as a background daemon, it runs in the root user context. So the $HOME for the SocketXP daemon process would be / or /root, depending on the Linux distribution. You could confirm this by executing sudo echo $HOME.

# SocketXP Log File

SocketXP IoT agent writes debug logs to a file located at $HOME/.socketxp/socketxp.log

SocketXP agent running in your device connects to the SocketXP IoT Cloud Gateway in the internet via your NAT router or firwall.

On successful connection, you'll see the following debug log in the socketxp.log file:

$ tail -f $HOME/.socketxp/socketxp.log

time="2021-04-09 16:30:10" level=debug msg="Starting SocketXP Client... Connecting local port 22 with SocketXP Cloud Gateway. hostname localhost https false\n\n"

Note:

SocketXP agent when run as a Linux Systemd service (background process), the SocketXP log file will be saved at: /.socketxp/socketxp.log or /root/.socketxp/socketxp.log

# SocketXP Systemd Logs

SocketXP single-touch installation script socketxp_install.sh installs and runs the SocketXP agent process as a Linux Systemd daemon.

SocketXP agent when run as a background process (as a Linux Systemd Daemon) will write to stdout and stderr which are stored as Systemd logs.

Check the SocketXP agent Systemd status using the following command:

$ systemctl status socketxp

● socketxp.service - SocketXP secure reverse proxy tunnels and webhook proxy service
     Loaded: loaded (/etc/systemd/system/socketxp.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2021-04-09 11:30:10 UTC; 12s ago
   Main PID: 85436 (socketxp)
      Tasks: 6 (limit: 9544)
     Memory: 4.4M
     CGroup: /system.slice/socketxp.service
             └─85436 /usr/local/bin/socketxp --config /root/config.json

Apr 09 11:30:10 socketxp systemd[1]: Started SocketXP secure reverse proxy tunnels and webhook proxy service.
Apr 09 11:30:10 socketxp socketxp[85436]: Using config file: /root/config.json
Apr 09 11:30:10 socketxp socketxp[85436]: Login Succeeded.
Apr 09 11:30:10 socketxp socketxp[85436]: User [] Email [[email protected]].
Apr 09 11:30:10 socketxp socketxp[85436]: Connected.
Apr 09 11:30:10 socketxp socketxp[85436]: TCP tunnel [test-user-sn5siexa] created.
Apr 09 11:30:10 socketxp socketxp[85436]: Access the tunnel using SocketXP agent in IoT Slave Mode.

SocketXP agent when run as a Linux Systemd service, the SocketXP log file will be saved at: /.socketxp/socketxp.log or /root/.socketxp/socketxp.log instead of saving it at: $HOME/.socketxp/socketxp

For Example:

$ cat /.socketxp/socketxp.log

time="2021-03-01 05:52:16" level=debug msg="Starting SocketXP Client... Connecting local port 22 with SocketXP Cloud Gateway. hostname 127.0.0.1 https false\n\n"