Command Line Kung Fu (2014)
Networking and SSH
Serve Files in the Current Directory via a Web Interface
$ python -m SimpleHTTPServer
$ python3 -m http.server
By default, this command starts a web server and serves up the content in the current directory over port 8000. You can change the port by specifying it at the end of the line. If no index.html file exists in the current directory, then the directory listing is shown. Start the web server and use a web browser to navigate to it. (firefox http://localhost:8000)
This can come in handy when you are working on HTML content and you want to see how it looks in a web browser without installing and configuring a full blown web server.
$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 …
localhost.localdomain - - [06/Apr/2014 21:49:20] "GET / HTTP/1.1" 200 -
Here's how to start the web server on the standard HTTP port. Since port 80 is a privileged port, IE it's 1024 or lower, doing this requires root privileges.
$ sudo python -m SimpleHTTPServer 80
Serving HTTP on 0.0.0.0 port 80 ...
Mount a Directory from a Remote Server on Your Local Host via SSH
$ sshfs remote-host:/directory mountpoint
$ fusermount -u mountpiont
Sometimes it's easier to work on files and directories if they are, or appear to be, local to your machine. For example, maybe you have a local application that doesn't exist on the server that you use to manipulate files. Instead of downloading the file from the server, modifying it, and and uploading it back to the server, you can mount the remote directory on your local workstation. Here is an example of updating a website over SSH.
$ mkdir web-files
$ sshfs www.example.com:/home/jason/public_html
$ bluefish web-files/index.html
$ fusermount -u web-files
Just like ssh command, you can use the user@host format if your remote username is different from your local username. Also, if no directory is specified after the colon, then your home directory is assumed.
Get Your Public IP from the Command Line Using Curl
$ curl ifconfig.me
If you ever need to determine your public (Internet) IP address you can use the ifconfig.me website.
$ curl ifconfig.me
198.145.20.140
$ curl ifconfig.me/ip
198.145.20.140
$ curl ifconfig.me/host
pub2.kernel.org
SSH into a Remote System without a Password
$ ssh-keygen
$ ssh-copy-id remote-host
$ ssh remote-host
In order to SSH into a remote host without a password you'll need an SSH key pair consisting of a private and public key. On the remote host the contents of the public key need to be in ~/.ssh/authorized_keys. The ssh-copy-id script performs that work.
If you want to generate a key without a password, simply hit enter when prompted for a passphrase. You can optionally supply a blank string to the -N option. (ssh-keygen -N '')
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/jason/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/jason/.ssh/id_rsa.
Your public key has been saved in /home/jason/.ssh/id_rsa.pub.
The key fingerprint is:
0d:2e:e4:32:dd:da:60:a5:2e:0f:c5:89:d5:78:30:ad jason@laptop.localdomain
The key's randomart image is:
+--[ RSA 2048]----+
| o. |
| =. |
| +.= |
| BEB o |
| + @ S . |
| * = |
| o o . |
| + |
| . |
+-----------------+
$ ssh-copy-id linuxserver
jason@192.168.122.60's password:
Now try logging into the machine, with "ssh 'linuxserver'", and check in:
.ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.
$ ssh linuxserver
$ hostname
linuxserver
$
Show Open Network Connections
$ sudo lsof -Pni
The lsof command can not only be used to display open files, but open network ports, and network connections. The -P option prevents the conversion of port numbers to port names. The -n option prevents the conversion of IP addresses to host names. The -i option tells lsof to display network connections.
$ sudo lsof -Pni
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
dhclient 989 root 6u IPv4 11522 0t0 UDP *:68
sshd 1202 root 3u IPv4 12418 0t0 TCP *:22 (LISTEN)
sshd 1202 root 4u IPv6 12423 0t0 TCP *:22 (LISTEN)
ntpd 1210 ntp 16u IPv4 12464 0t0 UDP *:123
ntpd 1210 ntp 17u IPv6 12465 0t0 UDP *:123
ntpd 1210 ntp 18u IPv4 12476 0t0 UDP 127.0.0.1:123
ntpd 1210 ntp 19u IPv4 12477 0t0 UDP 192.168.122.60:123
ntpd 1210 ntp 20u IPv6 12478 0t0 UDP [::1]:123
ntpd 1210 ntp 21u IPv6 12479 0t0 UDP [fe80::5054:ff:fe52:d858]:123
master 1364 root 12u IPv4 12761 0t0 TCP 127.0.0.1:25 (LISTEN)
clock-app 12174 jason 21u IPv4 78889 0t0 TCP 192.168.122.60:39021->184.25.102.40:80 (ESTABLISHED)
sshd 12339 root 3r IPv4 74023 0t0 TCP 192.168.122.60:22->192.168.122.1:34483 (ESTABLISHED)
sshd 12342 jason 3u IPv4 74023 0t0 TCP 192.168.122.60:22->192.168.122.1:34483 (ESTABLISHED)
$
Compare the Differences between a Remote and Local File
$ ssh remote-host cat /path/to/remotefile | diff /path/to/localfile -
To display the differences between a local and remote file, cat a file over ssh and pipe the output into a diff or sdiff command. The diff and sdiff commands can accept standard input in lieu of a file by supplying it a dash for one of the file names.
$ ssh linuxsvr cat /etc/passwd | diff /etc/passwd -
32c32
< terry:x:503:1000::/home/terry:/bin/ksh
---
> terry:x:503:1000::/home/terry:/bin/bash
35a36
> bob:x:1000:1000:Bob Smith:/home/bob:/bin/bash
$
Send Email from the Command Line
$ mail recipient@domain.com
$ echo 'message' | mail -s 'subject' recipient@domain.com
To send an email use the mail command. You can enter in a message interactively or via a pipe. End your interactive message with ctrl-d.
$ mail jim@mycorp.com
Subject: Message from the command line
Isn't this great?
EOT
$ echo "Here's the lazy way" | mail -s 'Message from the command line' jim@mycorp.com
Send an Email Attachment from the Command Line
$ mail -a /path/to/attachment
$ echo 'message' | mail -s 'subject' -a /path/to/attachment recipient@domain.com
If you ever need to send an email attachment from the command line, use the -a option to the mail command.
$ echo "Here is the file you requested" | mail -s "The file" -a /tmp/files.tgz jim@mycorp.com
$
Create an SSH Tunnel to Access Remote Resources
$ ssh -N -L local-port:host:remote-port remote-host
To create an SSH tunnel, use the -L option. The first port is the port that will be opened on your local machine. Connections to this port will be tunneled through remote-host and sent to the host and remote port specified in the -L option. The -N option tells SSH to not execute a command -- your shell -- on the remote host.
Let's say you want to access a website that isn't available on the internet, but is accessible from a server that you have SSH access to. You can create a tunnel that allows you to browse that website like you were behind the company's firewall. This command will forward any connections from your local machine on port 8000 through the jump server to the intranet server on port 80. Point your web browser to http://localhost:8000 and start surfing.
$ ssh -N -L 8000:intranet.acme.com:80 jump-server &
[1] 23253
$ firefox http://localhost:8000
Another use case is to access a service that is running on a server that you have SSH access to. If you need access to a mysql server that only allows database connections from specific hosts, you can create an SSH tunnel for your connection. Since the mysql service is running on localhost:3306 of the remote machine, the -L option would look like this: -L 3306:localhost:3306. You can use the mysql command line client on your local machine to connect to the database, but what's even more interesting is to use graphical desktop applications that aren't available on the server. For example, you could use this tunnel and connect to the database with MySQL Workbench, Navicat, or some other application.
$ ssh -N -L 3306:localhost:3306 db01 &
[1] 13455
$ mysql -h 127.0.0.1
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 5.1.73 Source distribution
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
Find out Which Programs Are Listening on Which Ports
$ sudo netstat -nutlp
Here are the descriptions of the netstat options used in order to get a list of programs and the ports that they are listening on.
-n show numerical addresses instead of determining symbolic names
-u include the UDP protocol
-t include the TCP protocol
-l show only listening sockets
-p show the PID and program name
$ sudo netstat -nutlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 4546/mysqld
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1161/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1325/master
tcp 0 0 :::80 :::* LISTEN 4576/httpd
tcp 0 0 :::22 :::* LISTEN 1161/sshd
udp 0 0 0.0.0.0:68 0.0.0.0:* 1008/dhclient
$
Use a Different SSH Key for a given Remote Host
Put the following in the ~/.ssh/config file.
Host remote-host
IdentityFile ~/.ssh/id_rsa-remote-host
If you need or want to use different SSH keys for different hosts, you can explicitly specify them on the command line with the -i option to ssh.
$ ssh -i ~/.ssh/id_rsa-db1 db1.example.com
If you want to forego specifying the key each time you can create an entry in your ~/.ssh/config file and specify the key there.
$ cat ~/.ssh/config
Host db1.example.com
IdentityFile ~/.ssh/id_rsa-db1
$ ssh db1.example.com
You can use wildcards in the host specification.
$ cat~/.ssh/config
Host db*
IdentityFile ~/.ssh/id_rsa-db1
Host *.work.net
IdentityFile ~/work-files/keys/id_rsa
$ ssh jim@jumpbox.work.net
If you name your SSH keys after the fully qualified domain names of the hosts they relate to, you can use the %h escape character to simplify your ~/.ssh/config file. Instead of having a host entry for each and every server, the %h syntax expands to the fully qualified domain name of the host your are connecting to.
$ cat ~/.ssh/config
Host *.example.com
IdentityFile ~/.ssh/id_rsa-%h
$ ls -1 ~/.ssh/id_rsa-*
id_rsa-lax-db-01.example.com
id_rsa-lax-db-01.example.com.pub
id_rsa-lax-web-01.example.com
id_rsa-lax-web-01.example.com.pub
$ ssh lax-db-01.example.com
Avoid Having to Type Your Username When Connecting via SSH
Put the following in the ~/.ssh/config file.
Host remote-host
User username
If you have a different username on your local Linux machine than you do on the remote linux machine, you have to specify it when connecting via SSH. It looks like this.
$ ssh jim@server1.example.com
To avoid having to type "username@" each time, add a host entry to your ~/.ssh/config file.
Host server1.example.com
User jim
Once your have configured the host entry, you can simply ssh into the remote host.
$ whoami
james
$ ssh server1.example.com
$ whoami
jim
$
Simplify Multi-Hop SSH Connections and Transparently Proxy SSH Connections
Put the following in the ~/.ssh/config file.
Host jumphost.example.com
ProxyCommand none
Host *.example.com
ProxyCommand ssh -W %h:%p jumphost.example.com
If you need to access a host that sits behind an SSH gateway server or jump server, you can make your life easier by telling SSH to automatically use the SSH gateway when you connect to the final remote host. Instead of first connecting to the gateway and then entering another ssh command to connect to the destination host, you simply type "ssh destination-host" from your local machine. Using the above configuration, this command will proxy your ssh connection to server1 through jumphost.
$ ssh server1.example.com
$ uname -n
server1
$
Disconnect from a Remote Session and Reconnect at a Later Time, Picking up Where You Left Off
$ ssh remote-host
$ screen
ctrl-a, d
$ exit
$ ssh remote-host
$ screen -r
When I have a long running process that I need to complete on a remote host, I always start a screen session before launching that process. I don't want a blip in my network connection to interrupt the work being performed on the remote host. Sometimes I launch a process, detach from the session, and reconnect later to examine all the output that occurred while I was away.
First, ssh into the remote host. Next, start a screen session. Start performing your work on the remote host. Detach from the screen session by typing ctrl-a followed by d. The process you started will still be running in the screen session while you're away. Also, any output generated will be available for you to view at a later time.
$ ssh remote-host
$ screen
$ /usr/local/bin/migrate-db
Starting DB migration at Sun Apr 13 21:02:50 EDT 2014
<ctrl-a,d>
[detached]
$ exit
To reconnect to your screen session, connect to the remote host and type screen -r. If there is any output that scrolled past the top of the screen, you can view by typing ctrl-a followed by the escape key. Now use the vi navigation key bindings to view the output history. For example, you can type k to move up one line or ctrl-b to page up. Once you are finished looking at the output history, hit escape to return to the live session. To quit your screen session, type exit.
$ ssh remote-host
$ screen -r
Starting DB migration at 21:02
table1 migrated at 21:34
table2 migrated at 22:11
table3 migrated at 22:54
DB migration completed at 23:04
$ exit
[screen is terminating]
$ exit
Screen is one of the most widely used and readily available screen multiplexers. However, there are alternatives such as tmux, dtach, and byobu.
Configure SSH to Append Domain Names to Host Names Based on a Pattern
The contents of ~/.ssh/config:
host-prefix* !*.domain.com
HostName %h.domain.com
If you connect to hosts in multiple domains via ssh it can get tiresome typing out the fully qualified domain name each time. One way around this problem is to add each domain to the search list in /etc/resolv.conf. The resolver will the attempt the resolution for the specified host name in each of the domains in the search list until it finds one that resolves.
$ cat /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
search domain1.com domain2.com domain3.com domain4.com domain5.com domain6.com domain7.com
When typing "ssh remote-host" with the above resolv.conf in place, the resolver will attempt to translate remote-host.domain1.com into an IP address. If that fails, it will attempt to resolve remote-host.domain2.com, etc. The problem with the above reslov.conf is that the search list is limited to just six domains. So, remote-host.domain7.com is never attempted. Additionally, the search list is limited to 256 characters, regardless the number of domains.
How can you get around the six domain search list limit? If you're lucky enough to have a pattern of hostnames that correlate with domain names, you can configure ssh to do the resolution. For example, for FQDNs like "ny-www1.newyork.company.com" and "ny-mysql-07.newyork.company.com" you can create a rule that appends ".newyork.company.com" to any host that begins with "ny." You'll also want to tell ssh to ignore any hosts that begin with "ny" that already have ".newyork.company.com" appended to them. Here's an example ~/.ssh/config file that does that.
$ cat ~/.ssh/config
ny* !*.newyork.company.com
HostName %h.newyork.company.com
db* !*.databases.company.com
HostName %h.databases.company.com
jump* !*.company.com
HostName %h.company.com
Now when you type "ssh ny-test" ssh will attempt to connect to "ny-test.newyork.company.com." For hosts that begin with "db," ssh will append ".databases.company.com" to the host name. Hosts the begin with "jump" will have the ".company.com" domain name appened to them.
$ ssh ny-www1
$ hostname -f
ny-www1.newyork.company.com
$ exit
$ ssh jump-ny-01
$ hostname -f
jump-ny-01.company.com
$ exit
$
Run a Command Immune to Hangups, Allowing the Job to Run after You Disconnect
$ nohup command &
Normally when you start a job in the background and log out of your session the job gets killed. One way to ensure a command keeps running after you disconnect from the host is to use the nohup command. No hup stands for no hang up. By default the output of the command is stored in a file named "nohup.out" in the directory the program was launched in. You can examine the contents of this file later to see the output of the command. To use a different filename, employ redirection.
$ ssh db-server
$ nohup /usr/local/bin/upgradedb.sh &
[1] 13370
$ exit
$ ssh db-server
$ cat nohup.out
Starting database upgrade.
...
Database upgrade complete.
$ nohup /usr/local/bin/post-upgrade.sh > /tmp/post.log &
[1] 16711
$ exit
$ ssh db-server
$ cat /tmp/post.log
Post processing completed.
$
Encrypt Your Web Browsing Data with an SSH SOCKS Proxy
$ ssh -D PORT remote-host
If you are using an open wireless hotspot and want to ensure your web browsing data is encrypted, you can redirect your web browsing traffic through another host via SSH. Start ssh with the "-D" option and provide a port to open up on your local computer for proxy connections. If you only want to perform the port forwarding and not actually log into the shell of the remote host, use the "-N" option for ssh. Configure your web browser to use a SOCKS 5 proxy using localhost for the host and the port you supplied to ssh.
$ ssh -ND 1080 ubuntu@ec2-75-101-157-145.compute-1.amazonaws.com
$ firefox http://www.mybank.com
Download a Webpage, HTTP Data, or Use a Web API from the Command Line
$ curl -o file.html http://website/webpage
$ wget http://website/webpage
The curl and wget commands can be used to download a webpage or anything that is available on a web server. You can use these commands to interact with HTTP APIs, download software packages, download a status page, or even get the current weather.
Here's an example of checking the status page of your local apache web server.
$ curl -o server-status.html http://localhost/server-status
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 6148 100 6148 0 0 1070k 0 --:--:-- --:--:-- --:--:-- 1200k
$ wget http://localhost/server-status
--2014-04-19 14:37:18-- http://localhost/server-status
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6377 (6.2K) [text/html]
Saving to: `server-status'
100%[=====>] 6,377 --.-K/s in 0s
2014-04-19 14:37:18 (105 MB/s) - `server-status' saved [6377/6377]
$ grep uptime server-status*
server-status:<dt>Server uptime: 50 minutes 13 seconds</dt>
server-status.html:<dt>Server uptime: 50 minutes 5 seconds</dt>
Here's an example of getting the current weather.
$ curl -so lax-weather.html http://weather.noaa.gov/pub/data/observations/metar/decoded/KLAX.TXT
$ cat lax-weather.html
LOS ANGELES INTERNTL AIRPORT, CA, United States (KLAX) 33-56N 118-23W 46M
Apr 19, 2014 - 02:53 PM EDT / 2014.04.19 1853 UTC
Wind: from the W (260 degrees) at 10 MPH (9 KT):0
Visibility: 10 mile(s):0
Sky conditions: mostly cloudy
Temperature: 64.9 F (18.3 C)
Dew Point: 54.0 F (12.2 C)
Relative Humidity: 67%
Pressure (altimeter): 30.03 in. Hg (1016 hPa)
ob: KLAX 191853Z 26009KT 10SM FEW022 BKN220 18/12 A3003 RMK AO2 SLP167 T01830122
cycle: 19
$ wget -q http://weather.noaa.gov/pub/data/observations/metar/decoded/KLAX.TXT
$ cat KLAX.TXT
LOS ANGELES INTERNTL AIRPORT, CA, United States (KLAX) 33-56N 118-23W 46M
Apr 19, 2014 - 02:53 PM EDT / 2014.04.19 1853 UTC
Wind: from the W (260 degrees) at 10 MPH (9 KT):0
Visibility: 10 mile(s):0
Sky conditions: mostly cloudy
Temperature: 64.9 F (18.3 C)
Dew Point: 54.0 F (12.2 C)
Relative Humidity: 67%
Pressure (altimeter): 30.03 in. Hg (1016 hPa)
ob: KLAX 191853Z 26009KT 10SM FEW022 BKN220 18/12 A3003 RMK AO2 SLP167 T01830122
cycle: 19
$
Download and install a package.
$ wget -q https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.1.deb
$ sudo dpkg -i elasticsearch-1.1.1.deb
Selecting previously unselected package elasticsearch.
(Reading database ... 162097 files and directories currently installed.)
Unpacking elasticsearch (from elasticsearch-1.1.1.deb) ...
Setting up elasticsearch (1.1.1) ...
Adding system user `elasticsearch' (UID 116) ...
Adding new user `elasticsearch' (UID 116) with group `elasticsearch' ...
Not creating home directory `/usr/share/elasticsearch'.
### NOT starting elasticsearch by default on bootup, please execute
sudo update-rc.d elasticsearch defaults 95 10
### In order to start elasticsearch, execute
sudo /etc/init.d/elasticsearch start
Processing triggers for ureadahead ...
$ sudo /etc/init.d/elasticsearch start
* Starting Elasticsearch Server [ OK ]
$
Interact with a web API.
$ curl http://localhost:9200
{
"status" : 200,
"name" : "NFL Superpro",
"version" : {
"number" : "1.1.1",
"build_hash" : "f1585f096d3f3985e73456debdc1a0745f512bbc",
"build_timestamp" : "2014-04-16T14:27:12Z",
"build_snapshot" : false,
"lucene_version" : "4.7"
},
"tagline" : "You Know, for Search"
}
$ curl http://localhost:9200/_cluster/health?pretty
{
"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}
$
Use Vim to Edit Files over the Network
$ vim scp://remote-host//path/to/file
$ vim scp://remote-user@remote-host//path/to/file
If you want to edit a file with vim over SSH, you can let it do the heavy lifting of copying the file back and forth.
$ vim scp://linuxserver//home/jason/notes.txt