How to Debug and Troubleshoot SocketXP
Here in this section, you'll find all the information you need to debug and troubleshoot any problem with the SocketXP agent.
Cannot execute binary file: Exec format error
When you execute the SocketXP agent on your device and if you get the following error:
Error
$ sudo socketxp
cannot execute binary file: Exec format error.
It means you have downloaded the wrong binary for your hardware platform and OS type.
You'll get this error if you try to run, for example, a Linux binary on a MacOS or a 64-bit Linux ARM binary on a 32-bit or 64-bit ARM processor running 32-bit Linux OS.
SocketXP Config File Location
If you used the single-touch installation script socketxp_install.sh to install and configure SocketXP IoT agent in your IoT or Raspberry Pi device, you can find the configuration file at /etc/socketxp/config.json.
A sample config.json file would look like the following:
$ cat /etc/socketxp/config.json
{
"iot_device_name": "device12345",
"iot_device_group": "temp-sensor",
"tunnels" : [
{
"destination": "tcp://127.0.0.1:22"
}
]
}
The config.json file contents are in JSON file format. More information about the contents of the config.json file can be found here.
Note
iot_device_name and iot_device_group fields can be set once during the initial device setup. Thereafter if you want to change it, you cannot do so from the config.json file. You need to edit the device in the portal page and update its name, group, customer name and location information and save it.
SocketXP Device Key File Location:
SocketXP installs a per device private key file at: /var/lib/socketxp/device.key when you login to the SocketXP Cloud Gateway using the SocketXP login command. This key is used to authenticate the device with the user account. The key is also used to uniquely identify the device and the services running in the device that needs remote access.
If you plan to clone your IoT device OS image into multiple SSD or flash cards, make sure you delete this device.key file before cloning.
If you delete a device from the Devices section in the SocketXP Portal page, you also need to delete the /var/lib/socketxp/device.key file in the device before you could add the device back to your account. Otherwise you'll get the following error on your console:
Server Error: Invalid device key (REclP7PzXMTe61zh3ASa0jd7Y0k3AiF...). Delete device key file at /var/lib/socketxp/device.key and then login again.
Invalid Device Key Error (EU Region Users)
Our default gateway is US region gateway. If you are a user in the Europe region and trying to register/connect your device with the EU gateway, you may get the following error if you fail to include the --region "eu" flag in the "socketxp login" or "socketxp connect" or any socketxp command.
Server Error: Invalid device key (REclP7PzXMTe61zh3ASa0jd7Y0k3AiF...). Delete device key file at /var/lib/socketxp/device.key and then login again.
More info about the region flag can be found here: SocketXP Gateway Region
The above error will occur if you used --region "eu" option in the socketxp login command but failed to include the --region "eu" option in the socketxp connect command. By default, all socketxp commands will try communicating with the US gateway, which is the default gateway.
To resolve the error, run the socketxp connect command as follows:
Note
You cannot register/login your device with one gateway(EU, for example) and try connecting it to another gateway(US gateway, for example). You cannot use the credentials obtained from a gateway in one region to connect with a gateway in another region.
Credentials are NOT global objects. Credentials are local to the gateway. Credentials include authtoken and device key obtained from the gateway's web portal.
SocketXP Log File
SocketXP IoT agent writes debug logs to a file located at /var/lib/socketxp/socketxp.log
SocketXP agent running in your device connects to the SocketXP IoT Cloud Gateway in the internet via your NAT router or firwall.
On successful connection, you'll see the following debug log in the socketxp.log file:
$ tail -f /var/lib/socketxp/socketxp.log
time="2021-04-09 16:30:10" level=debug msg="Starting SocketXP Client... Connecting local port 22 with SocketXP Cloud Gateway. hostname localhost https false\n\n"
SocketXP Systemd Logs
SocketXP single-touch installation script socketxp_install.sh installs and runs the SocketXP agent process as a Linux Systemd daemon in the background.
SocketXP agent when run as a background process (as a Linux Systemd Daemon) will write to stdout and stderr which are stored as Systemd logs.
Check the SocketXP agent Systemd status and logs using the following command:
$ systemctl status socketxp
● socketxp.service - SocketXP secure reverse proxy tunnels and webhook proxy service
Loaded: loaded (/etc/systemd/system/socketxp.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2021-04-09 11:30:10 UTC; 12s ago
Main PID: 85436 (socketxp)
Tasks: 6 (limit: 9544)
Memory: 4.4M
CGroup: /system.slice/socketxp.service
└─85436 /usr/local/bin/socketxp --config /root/config.json
Apr 09 11:30:10 socketxp systemd[1]: Started SocketXP secure reverse proxy tunnels and webhook proxy service.
Apr 09 11:30:10 socketxp socketxp[85436]: Using config file: /root/config.json
Apr 09 11:30:10 socketxp socketxp[85436]: Login Succeeded.
Apr 09 11:30:10 socketxp socketxp[85436]: User [] Email [[email protected]].
Apr 09 11:30:10 socketxp socketxp[85436]: Connected.
Apr 09 11:30:10 socketxp socketxp[85436]: TCP tunnel [test-user-sn5siexa] created.
Apr 09 11:30:10 socketxp socketxp[85436]: Access the tunnel using SocketXP agent in IoT Slave Mode.
SocketXP Syslogs
SocketXP when run as a Systemd service also logs to the /var/log/syslog file. All systemd services running in your device would log to the same syslog file. You could use the below command to grep only the SocketXP syslogs and save it in a separate file.
A sample SocketXP syslog content might look like the following:
Apr 18 04:30:26 Ubuntu systemd[1]: socketxp.service: Main process exited, code=exited, status=255/n/a
Apr 18 04:30:26 Ubuntu systemd[1]: socketxp.service: Failed with result 'exit-code'.
Apr 18 04:30:31 Ubuntu systemd[1]: socketxp.service: Service hold-off time over, scheduling restart.
Apr 18 04:30:31 Ubuntu systemd[1]: socketxp.service: Scheduled restart job, restart counter is at 29.
Apr 18 04:30:31 Ubuntu socketxp[16506]: Using config file: /home/test-user/config.json
Apr 18 04:30:31 Ubuntu socketxp[16506]: Login Succeeded.
Apr 18 04:30:31 Ubuntu socketxp[16506]: User [] Email [[email protected]].
Apr 18 04:30:31 Ubuntu socketxp[16506]: Could not create service endpoint in SocketXP Cloud Service. Server Error: Device ID already exists; please enter a unique value for device ID.
Apr 18 04:30:31 Ubuntu systemd[1]: socketxp.service: Main process exited, code=exited, status=255/n/a
Apr 18 04:30:31 Ubuntu systemd[1]: socketxp.service: Failed with result 'exit-code'.
Apr 18 04:30:36 Ubuntu systemd[1]: socketxp.service: Service hold-off time over, scheduling restart.
Apr 18 04:30:36 Ubuntu systemd[1]: socketxp.service: Scheduled restart job, restart counter is at 30.
Apr 18 04:30:36 Ubuntu socketxp[16578]: Using config file: /home/test-user/config.json
Apr 18 04:30:36 Ubuntu socketxp[16578]: Login Succeeded.
Apr 18 04:30:36 Ubuntu socketxp[16578]: User [] Email [[email protected]].
Apr 18 04:30:36 Ubuntu socketxp[16578]: Connected.
Apr 18 04:30:36 Ubuntu socketxp[16578]: TCP tunnel [test-user-i3d6l5wn] created.
Apr 18 04:30:36 Ubuntu socketxp[16578]: Access the tunnel using SocketXP agent in IoT Slave Mode.
SocketXP Linux Systemd Config File
You can find the SocketXP Linux Systemd service configuration file at the following location with the following contents:
$ cat /etc/systemd/system/socketxp.service
[Unit]
Description=SocketXP secure IoT remote access service
Requires=network.target
After=network.target
[Service]
Type=simple
Restart=always
RestartSec=5
ExecStart=/usr/local/bin/socketxp --config /etc/socketxp/config.json
ExecStop=/bin/kill -15 $MAINPID
KillMode=process
[Install]
WantedBy=multi-user.target
SocketXP Login Error
When you execute the socketxp login command and you get the following error:
$ socketxp login "eyJhbGciOiJIUzI1NiIsInR5c..."
Could not connect to SocketXP Cloud Gateway. Please check internet connectivity or SocketXP Cloud Gateway availability.
- Your internet connection could be down
- Your corporate firewall or proxy server settings is very restrictive and is preventing SocketXP IoT agent (running in your IoT device behind NAT router or Firewall) from reaching the internet. Read the section on Whitelisting SocketXP Service (see below) to make the necessary changes in your firewall or proxy server setttings for SocketXP to work. Most users won't have this problem.
- SocketXP IoT Cloud Gateway could be extremely busy(a rare event) and may not be accepting new device registrations temporarily. You may have to retry after few minutes.
- SocketXP IoT Cloud Gateway could be down (which is very very highly unlikely). We strive to keep our service up and runing 100% of the time: 24x7 and 365 days a year. However, SocketXP IoT Cloud Gateway could be down for a brief maintenance. Please note that SocketXP Cloud Gateway will be down for a brief maintenance between the hours of 9:00 PM PST to 12:00 PM PST on Saturdays/Sundays only.
Whitelisting SocketXP Service
Some corporate networks have a very restrictive firewall or a proxy server that allows outbound traffic to certain domains in the internet and on specific TCP or UDP ports. Outbound traffic from the corporate network to all others domains in the internet, including *.socketxp.com will be restricted. In such situations, you need to request your IT team to add SocketXP service to your corporate firewall's or proxy server's whitelist.
We do not recommend whitelisting our SocketXP IoT Cloud Gateway server's public IP address in your firewall settings. Instead, we recommend you to whitelist our wildcard domain name *.socketxp.com and permit all outbound traffic for TCP ports 9443 and 443 in your firewall settings.
For Legacy Agents - Version 1.4.5 and earlier
Add firewall rules to allow outbound traffic on TCP ports 80 and 443 for all legacy SocketXP agent versions 1.4.5 and earlier.
SocketXP agent communicates with the Cloud Gateway using secure TLS tunnels.
SocketXP Cloud Gateway will never establish any inbound connections to your device behind the firewall. So no modification is required in your inbound firewall rules. SocketXP agent will establish only outbound connections to *.socketxp.com domain in the internet on TCP ports 9443 and 443.
Error opening vi or vim editor in the SSH session started from the web terminal
Sometimes, opening the vi or vim editor in the SSH session started from the web terminal may close the session abruptly. All other Linux commands may work just fine. Even, opening a Nano editor or any other editor may work just fine.
The fix to address this issue is to check the locale settings in your shell's environment variable. Set the following settings in your shell's environment:
Also, add the above lines to your ~/.bashrc file or to your preferred shell's rc file.
Troubleshooting Legacy SocketXP Agents
In the below sections, you'll find the instructions to troubleshoot SocketXP Agent version 1.3.0 and earlier versions.
SocketXP Config File Location
If you used the single-touch installation script socketxp_install.sh to install and configure SocketXP IoT agent in your IoT or Raspberry Pi device, you can find the configuration file at /config.json or /root/config.json
A sample legacy config.json file would look like the following:
$ cat $HOME/config.json
{
"authtoken": "<your-auth-token-here>",
"tunnel_enabled": true,
"tunnels" : [
{
"destination": "tcp://127.0.0.1:22",
"protocol": "tcp",
"custom_domain": "",
"subdomain": "",
"iot_device_id": "abc123456789",
"iot_device_name": "some_unique_device_name",
"iot_device_group": "some_device_grouping"
}
],
"relay_enabled": false,
"relays": [{ }]
}
The config.json file contents are in JSON file format. More information about the contents of the config.json file can be found here.
SocketXP Device Key File Location:
SocketXP installs a per device private key at: $HOME/.socketxp/device.key. This key is used to authenticate the device with the user account. The key is also used to uniquely identify the device and the services running in the device.
If you delete a device from the Devices section in the SocketXP Portal page, you need to delete the $HOME/.socketxp/device.key file also before you add the device back to your account. Otherwise you'll get an error as shown below:
Server Error: Invalid device key (REclP7PzXMTe61zh3ASa0jd7Y0k3AiF...). Delete device key file at $HOME/.socketxp/device.key and re-login.
Note
Please confirm your $HOME by executing echo $HOME before you use it. When you run SocketXP agent as a background daemon, it runs in the root user context. So the $HOME for the SocketXP daemon process would be / or /root, depending on the Linux distribution. You could confirm this by executing sudo echo $HOME.
SocketXP Log File
SocketXP IoT agent writes debug logs to a file located at $HOME/.socketxp/socketxp.log
SocketXP agent running in your device connects to the SocketXP IoT Cloud Gateway in the internet via your NAT router or firwall.
On successful connection, you'll see the following debug log in the socketxp.log file:
$ tail -f $HOME/.socketxp/socketxp.log
time="2021-04-09 16:30:10" level=debug msg="Starting SocketXP Client... Connecting local port 22 with SocketXP Cloud Gateway. hostname localhost https false\n\n"
Note
SocketXP agent when run as a Linux Systemd service (background process), the SocketXP log file will be saved at: /.socketxp/socketxp.log or /root/.socketxp/socketxp.log
SocketXP Systemd Logs
SocketXP single-touch installation script socketxp_install.sh installs and runs the SocketXP agent process as a Linux Systemd daemon.
SocketXP agent when run as a background process (as a Linux Systemd Daemon) will write to stdout and stderr which are stored as Systemd logs.
Check the SocketXP agent Systemd status using the following command:
$ systemctl status socketxp
● socketxp.service - SocketXP remote access service
Loaded: loaded (/etc/systemd/system/socketxp.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2021-04-09 11:30:10 UTC; 12s ago
Main PID: 85436 (socketxp)
Tasks: 6 (limit: 9544)
Memory: 4.4M
CGroup: /system.slice/socketxp.service
└─85436 /usr/local/bin/socketxp --config /root/config.json
Apr 09 11:30:10 socketxp systemd[1]: Started SocketXP remote access service.
Apr 09 11:30:10 socketxp socketxp[85436]: Using config file: /root/config.json
Apr 09 11:30:10 socketxp socketxp[85436]: Login Succeeded.
Apr 09 11:30:10 socketxp socketxp[85436]: User [] Email [[email protected]].
Apr 09 11:30:10 socketxp socketxp[85436]: Connected.
Apr 09 11:30:10 socketxp socketxp[85436]: TCP tunnel [test-user-sn5siexa] created.
Apr 09 11:30:10 socketxp socketxp[85436]: Access the tunnel using SocketXP agent in IoT Slave Mode.
SocketXP agent when run as a Linux Systemd service, the SocketXP log file will be saved at: /.socketxp/socketxp.log or /root/.socketxp/socketxp.log instead of saving it at: $HOME/.socketxp/socketxp
For Example: