Networking and SSH - Command Line Kung Fu (2014)

Command Line Kung Fu (2014)

Networking and SSH

Serve Files in the Current Directory via a Web Interface

$ python -m SimpleHTTPServer

$ python3 -m http.server

By default, this command starts a web server and serves up the content in the current directory over port 8000. You can change the port by specifying it at the end of the line. If no index.html file exists in the current directory, then the directory listing is shown. Start the web server and use a web browser to navigate to it. (firefox http://localhost:8000)

This can come in handy when you are working on HTML content and you want to see how it looks in a web browser without installing and configuring a full blown web server.

$ python -m SimpleHTTPServer

Serving HTTP on 0.0.0.0 port 8000 …

localhost.localdomain - - [06/Apr/2014 21:49:20] "GET / HTTP/1.1" 200 -

Here's how to start the web server on the standard HTTP port. Since port 80 is a privileged port, IE it's 1024 or lower, doing this requires root privileges.

$ sudo python -m SimpleHTTPServer 80

Serving HTTP on 0.0.0.0 port 80 ...

Mount a Directory from a Remote Server on Your Local Host via SSH

$ sshfs remote-host:/directory mountpoint

$ fusermount -u mountpiont

Sometimes it's easier to work on files and directories if they are, or appear to be, local to your machine. For example, maybe you have a local application that doesn't exist on the server that you use to manipulate files. Instead of downloading the file from the server, modifying it, and and uploading it back to the server, you can mount the remote directory on your local workstation. Here is an example of updating a website over SSH.

$ mkdir web-files

$ sshfs www.example.com:/home/jason/public_html

$ bluefish web-files/index.html

$ fusermount -u web-files

Just like ssh command, you can use the user@host format if your remote username is different from your local username. Also, if no directory is specified after the colon, then your home directory is assumed.

Get Your Public IP from the Command Line Using Curl

$ curl ifconfig.me

If you ever need to determine your public (Internet) IP address you can use the ifconfig.me website.

$ curl ifconfig.me

198.145.20.140

$ curl ifconfig.me/ip

198.145.20.140

$ curl ifconfig.me/host

pub2.kernel.org

SSH into a Remote System without a Password

$ ssh-keygen

$ ssh-copy-id remote-host

$ ssh remote-host

In order to SSH into a remote host without a password you'll need an SSH key pair consisting of a private and public key. On the remote host the contents of the public key need to be in ~/.ssh/authorized_keys. The ssh-copy-id script performs that work.

If you want to generate a key without a password, simply hit enter when prompted for a passphrase. You can optionally supply a blank string to the -N option. (ssh-keygen -N '')

$ ssh-keygen

Generating public/private rsa key pair.

Enter file in which to save the key (/home/jason/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/jason/.ssh/id_rsa.

Your public key has been saved in /home/jason/.ssh/id_rsa.pub.

The key fingerprint is:

0d:2e:e4:32:dd:da:60:a5:2e:0f:c5:89:d5:78:30:ad jason@laptop.localdomain

The key's randomart image is:

+--[ RSA 2048]----+

| o. |

| =. |

| +.= |

| BEB o |

| + @ S . |

| * = |

| o o . |

| + |

| . |

+-----------------+

$ ssh-copy-id linuxserver

jason@192.168.122.60's password:

Now try logging into the machine, with "ssh 'linuxserver'", and check in:

.ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

$ ssh linuxserver

$ hostname

linuxserver

$

Show Open Network Connections

$ sudo lsof -Pni

The lsof command can not only be used to display open files, but open network ports, and network connections. The -P option prevents the conversion of port numbers to port names. The -n option prevents the conversion of IP addresses to host names. The -i option tells lsof to display network connections.

$ sudo lsof -Pni

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

dhclient 989 root 6u IPv4 11522 0t0 UDP *:68

sshd 1202 root 3u IPv4 12418 0t0 TCP *:22 (LISTEN)

sshd 1202 root 4u IPv6 12423 0t0 TCP *:22 (LISTEN)

ntpd 1210 ntp 16u IPv4 12464 0t0 UDP *:123

ntpd 1210 ntp 17u IPv6 12465 0t0 UDP *:123

ntpd 1210 ntp 18u IPv4 12476 0t0 UDP 127.0.0.1:123

ntpd 1210 ntp 19u IPv4 12477 0t0 UDP 192.168.122.60:123

ntpd 1210 ntp 20u IPv6 12478 0t0 UDP [::1]:123

ntpd 1210 ntp 21u IPv6 12479 0t0 UDP [fe80::5054:ff:fe52:d858]:123

master 1364 root 12u IPv4 12761 0t0 TCP 127.0.0.1:25 (LISTEN)

clock-app 12174 jason 21u IPv4 78889 0t0 TCP 192.168.122.60:39021->184.25.102.40:80 (ESTABLISHED)

sshd 12339 root 3r IPv4 74023 0t0 TCP 192.168.122.60:22->192.168.122.1:34483 (ESTABLISHED)

sshd 12342 jason 3u IPv4 74023 0t0 TCP 192.168.122.60:22->192.168.122.1:34483 (ESTABLISHED)

$

Compare the Differences between a Remote and Local File

$ ssh remote-host cat /path/to/remotefile | diff /path/to/localfile -

To display the differences between a local and remote file, cat a file over ssh and pipe the output into a diff or sdiff command. The diff and sdiff commands can accept standard input in lieu of a file by supplying it a dash for one of the file names.

$ ssh linuxsvr cat /etc/passwd | diff /etc/passwd -

32c32

< terry:x:503:1000::/home/terry:/bin/ksh

---

> terry:x:503:1000::/home/terry:/bin/bash

35a36

> bob:x:1000:1000:Bob Smith:/home/bob:/bin/bash

$

Send Email from the Command Line

$ mail recipient@domain.com

$ echo 'message' | mail -s 'subject' recipient@domain.com

To send an email use the mail command. You can enter in a message interactively or via a pipe. End your interactive message with ctrl-d.

$ mail jim@mycorp.com

Subject: Message from the command line

Isn't this great?

EOT

$ echo "Here's the lazy way" | mail -s 'Message from the command line' jim@mycorp.com

Send an Email Attachment from the Command Line

$ mail -a /path/to/attachment

$ echo 'message' | mail -s 'subject' -a /path/to/attachment recipient@domain.com

If you ever need to send an email attachment from the command line, use the -a option to the mail command.

$ echo "Here is the file you requested" | mail -s "The file" -a /tmp/files.tgz jim@mycorp.com

$

Create an SSH Tunnel to Access Remote Resources

$ ssh -N -L local-port:host:remote-port remote-host

To create an SSH tunnel, use the -L option. The first port is the port that will be opened on your local machine. Connections to this port will be tunneled through remote-host and sent to the host and remote port specified in the -L option. The -N option tells SSH to not execute a command -- your shell -- on the remote host.

Let's say you want to access a website that isn't available on the internet, but is accessible from a server that you have SSH access to. You can create a tunnel that allows you to browse that website like you were behind the company's firewall. This command will forward any connections from your local machine on port 8000 through the jump server to the intranet server on port 80. Point your web browser to http://localhost:8000 and start surfing.

$ ssh -N -L 8000:intranet.acme.com:80 jump-server &

[1] 23253

$ firefox http://localhost:8000

Another use case is to access a service that is running on a server that you have SSH access to. If you need access to a mysql server that only allows database connections from specific hosts, you can create an SSH tunnel for your connection. Since the mysql service is running on localhost:3306 of the remote machine, the -L option would look like this: -L 3306:localhost:3306. You can use the mysql command line client on your local machine to connect to the database, but what's even more interesting is to use graphical desktop applications that aren't available on the server. For example, you could use this tunnel and connect to the database with MySQL Workbench, Navicat, or some other application.

$ ssh -N -L 3306:localhost:3306 db01 &

[1] 13455

$ mysql -h 127.0.0.1

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connection id is 9

Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

Find out Which Programs Are Listening on Which Ports

$ sudo netstat -nutlp

Here are the descriptions of the netstat options used in order to get a list of programs and the ports that they are listening on.

-n show numerical addresses instead of determining symbolic names

-u include the UDP protocol

-t include the TCP protocol

-l show only listening sockets

-p show the PID and program name

$ sudo netstat -nutlp

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 4546/mysqld

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1161/sshd

tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1325/master

tcp 0 0 :::80 :::* LISTEN 4576/httpd

tcp 0 0 :::22 :::* LISTEN 1161/sshd

udp 0 0 0.0.0.0:68 0.0.0.0:* 1008/dhclient

$

Use a Different SSH Key for a given Remote Host

Put the following in the ~/.ssh/config file.

Host remote-host

IdentityFile ~/.ssh/id_rsa-remote-host

If you need or want to use different SSH keys for different hosts, you can explicitly specify them on the command line with the -i option to ssh.

$ ssh -i ~/.ssh/id_rsa-db1 db1.example.com

If you want to forego specifying the key each time you can create an entry in your ~/.ssh/config file and specify the key there.

$ cat ~/.ssh/config

Host db1.example.com

IdentityFile ~/.ssh/id_rsa-db1

$ ssh db1.example.com

You can use wildcards in the host specification.

$ cat~/.ssh/config

Host db*

IdentityFile ~/.ssh/id_rsa-db1

Host *.work.net

IdentityFile ~/work-files/keys/id_rsa

$ ssh jim@jumpbox.work.net

If you name your SSH keys after the fully qualified domain names of the hosts they relate to, you can use the %h escape character to simplify your ~/.ssh/config file. Instead of having a host entry for each and every server, the %h syntax expands to the fully qualified domain name of the host your are connecting to.

$ cat ~/.ssh/config

Host *.example.com

IdentityFile ~/.ssh/id_rsa-%h

$ ls -1 ~/.ssh/id_rsa-*

id_rsa-lax-db-01.example.com

id_rsa-lax-db-01.example.com.pub

id_rsa-lax-web-01.example.com

id_rsa-lax-web-01.example.com.pub

$ ssh lax-db-01.example.com

Avoid Having to Type Your Username When Connecting via SSH

Put the following in the ~/.ssh/config file.

Host remote-host

User username

If you have a different username on your local Linux machine than you do on the remote linux machine, you have to specify it when connecting via SSH. It looks like this.

$ ssh jim@server1.example.com

To avoid having to type "username@" each time, add a host entry to your ~/.ssh/config file.

Host server1.example.com

User jim

Once your have configured the host entry, you can simply ssh into the remote host.

$ whoami

james

$ ssh server1.example.com

$ whoami

jim

$

Simplify Multi-Hop SSH Connections and Transparently Proxy SSH Connections

Put the following in the ~/.ssh/config file.

Host jumphost.example.com

ProxyCommand none

Host *.example.com

ProxyCommand ssh -W %h:%p jumphost.example.com

If you need to access a host that sits behind an SSH gateway server or jump server, you can make your life easier by telling SSH to automatically use the SSH gateway when you connect to the final remote host. Instead of first connecting to the gateway and then entering another ssh command to connect to the destination host, you simply type "ssh destination-host" from your local machine. Using the above configuration, this command will proxy your ssh connection to server1 through jumphost.

$ ssh server1.example.com
$ uname -n

server1
$

Disconnect from a Remote Session and Reconnect at a Later Time, Picking up Where You Left Off

$ ssh remote-host

$ screen

ctrl-a, d

$ exit

$ ssh remote-host

$ screen -r

When I have a long running process that I need to complete on a remote host, I always start a screen session before launching that process. I don't want a blip in my network connection to interrupt the work being performed on the remote host. Sometimes I launch a process, detach from the session, and reconnect later to examine all the output that occurred while I was away.

First, ssh into the remote host. Next, start a screen session. Start performing your work on the remote host. Detach from the screen session by typing ctrl-a followed by d. The process you started will still be running in the screen session while you're away. Also, any output generated will be available for you to view at a later time.

$ ssh remote-host

$ screen

$ /usr/local/bin/migrate-db

Starting DB migration at Sun Apr 13 21:02:50 EDT 2014

<ctrl-a,d>

[detached]

$ exit

To reconnect to your screen session, connect to the remote host and type screen -r. If there is any output that scrolled past the top of the screen, you can view by typing ctrl-a followed by the escape key. Now use the vi navigation key bindings to view the output history. For example, you can type k to move up one line or ctrl-b to page up. Once you are finished looking at the output history, hit escape to return to the live session. To quit your screen session, type exit.

$ ssh remote-host

$ screen -r

Starting DB migration at 21:02

table1 migrated at 21:34

table2 migrated at 22:11

table3 migrated at 22:54

DB migration completed at 23:04

$ exit

[screen is terminating]

$ exit

Screen is one of the most widely used and readily available screen multiplexers. However, there are alternatives such as tmux, dtach, and byobu.

Configure SSH to Append Domain Names to Host Names Based on a Pattern

The contents of ~/.ssh/config:

host-prefix* !*.domain.com

HostName %h.domain.com

If you connect to hosts in multiple domains via ssh it can get tiresome typing out the fully qualified domain name each time. One way around this problem is to add each domain to the search list in /etc/resolv.conf. The resolver will the attempt the resolution for the specified host name in each of the domains in the search list until it finds one that resolves.

$ cat /etc/resolv.conf

nameserver 8.8.8.8

nameserver 8.8.4.4

search domain1.com domain2.com domain3.com domain4.com domain5.com domain6.com domain7.com

When typing "ssh remote-host" with the above resolv.conf in place, the resolver will attempt to translate remote-host.domain1.com into an IP address. If that fails, it will attempt to resolve remote-host.domain2.com, etc. The problem with the above reslov.conf is that the search list is limited to just six domains. So, remote-host.domain7.com is never attempted. Additionally, the search list is limited to 256 characters, regardless the number of domains.

How can you get around the six domain search list limit? If you're lucky enough to have a pattern of hostnames that correlate with domain names, you can configure ssh to do the resolution. For example, for FQDNs like "ny-www1.newyork.company.com" and "ny-mysql-07.newyork.company.com" you can create a rule that appends ".newyork.company.com" to any host that begins with "ny." You'll also want to tell ssh to ignore any hosts that begin with "ny" that already have ".newyork.company.com" appended to them. Here's an example ~/.ssh/config file that does that.

$ cat ~/.ssh/config

ny* !*.newyork.company.com

HostName %h.newyork.company.com

db* !*.databases.company.com

HostName %h.databases.company.com

jump* !*.company.com

HostName %h.company.com

Now when you type "ssh ny-test" ssh will attempt to connect to "ny-test.newyork.company.com." For hosts that begin with "db," ssh will append ".databases.company.com" to the host name. Hosts the begin with "jump" will have the ".company.com" domain name appened to them.

$ ssh ny-www1

$ hostname -f
ny-www1.newyork.company.com

$ exit

$ ssh jump-ny-01

$ hostname -f

jump-ny-01.company.com

$ exit

$

Run a Command Immune to Hangups, Allowing the Job to Run after You Disconnect

$ nohup command &

Normally when you start a job in the background and log out of your session the job gets killed. One way to ensure a command keeps running after you disconnect from the host is to use the nohup command. No hup stands for no hang up. By default the output of the command is stored in a file named "nohup.out" in the directory the program was launched in. You can examine the contents of this file later to see the output of the command. To use a different filename, employ redirection.

$ ssh db-server

$ nohup /usr/local/bin/upgradedb.sh &

[1] 13370

$ exit

$ ssh db-server

$ cat nohup.out

Starting database upgrade.

...

Database upgrade complete.

$ nohup /usr/local/bin/post-upgrade.sh > /tmp/post.log &

[1] 16711

$ exit

$ ssh db-server

$ cat /tmp/post.log

Post processing completed.

$

Encrypt Your Web Browsing Data with an SSH SOCKS Proxy

$ ssh -D PORT remote-host

If you are using an open wireless hotspot and want to ensure your web browsing data is encrypted, you can redirect your web browsing traffic through another host via SSH. Start ssh with the "-D" option and provide a port to open up on your local computer for proxy connections. If you only want to perform the port forwarding and not actually log into the shell of the remote host, use the "-N" option for ssh. Configure your web browser to use a SOCKS 5 proxy using localhost for the host and the port you supplied to ssh.

$ ssh -ND 1080 ubuntu@ec2-75-101-157-145.compute-1.amazonaws.com

$ firefox http://www.mybank.com

Download a Webpage, HTTP Data, or Use a Web API from the Command Line

$ curl -o file.html http://website/webpage

$ wget http://website/webpage

The curl and wget commands can be used to download a webpage or anything that is available on a web server. You can use these commands to interact with HTTP APIs, download software packages, download a status page, or even get the current weather.

Here's an example of checking the status page of your local apache web server.

$ curl -o server-status.html http://localhost/server-status

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 6148 100 6148 0 0 1070k 0 --:--:-- --:--:-- --:--:-- 1200k

$ wget http://localhost/server-status

--2014-04-19 14:37:18-- http://localhost/server-status

Resolving localhost (localhost)... 127.0.0.1

Connecting to localhost (localhost)|127.0.0.1|:80... connected.

HTTP request sent, awaiting response... 200 OK

Length: 6377 (6.2K) [text/html]

Saving to: `server-status'

100%[=====>] 6,377 --.-K/s in 0s

2014-04-19 14:37:18 (105 MB/s) - `server-status' saved [6377/6377]

$ grep uptime server-status*

server-status:<dt>Server uptime: 50 minutes 13 seconds</dt>

server-status.html:<dt>Server uptime: 50 minutes 5 seconds</dt>

Here's an example of getting the current weather.

$ curl -so lax-weather.html http://weather.noaa.gov/pub/data/observations/metar/decoded/KLAX.TXT

$ cat lax-weather.html

LOS ANGELES INTERNTL AIRPORT, CA, United States (KLAX) 33-56N 118-23W 46M

Apr 19, 2014 - 02:53 PM EDT / 2014.04.19 1853 UTC

Wind: from the W (260 degrees) at 10 MPH (9 KT):0

Visibility: 10 mile(s):0

Sky conditions: mostly cloudy

Temperature: 64.9 F (18.3 C)

Dew Point: 54.0 F (12.2 C)

Relative Humidity: 67%

Pressure (altimeter): 30.03 in. Hg (1016 hPa)

ob: KLAX 191853Z 26009KT 10SM FEW022 BKN220 18/12 A3003 RMK AO2 SLP167 T01830122

cycle: 19

$ wget -q http://weather.noaa.gov/pub/data/observations/metar/decoded/KLAX.TXT

$ cat KLAX.TXT

LOS ANGELES INTERNTL AIRPORT, CA, United States (KLAX) 33-56N 118-23W 46M

Apr 19, 2014 - 02:53 PM EDT / 2014.04.19 1853 UTC

Wind: from the W (260 degrees) at 10 MPH (9 KT):0

Visibility: 10 mile(s):0

Sky conditions: mostly cloudy

Temperature: 64.9 F (18.3 C)

Dew Point: 54.0 F (12.2 C)

Relative Humidity: 67%

Pressure (altimeter): 30.03 in. Hg (1016 hPa)

ob: KLAX 191853Z 26009KT 10SM FEW022 BKN220 18/12 A3003 RMK AO2 SLP167 T01830122

cycle: 19

$

Download and install a package.

$ wget -q https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.1.deb

$ sudo dpkg -i elasticsearch-1.1.1.deb

Selecting previously unselected package elasticsearch.

(Reading database ... 162097 files and directories currently installed.)

Unpacking elasticsearch (from elasticsearch-1.1.1.deb) ...

Setting up elasticsearch (1.1.1) ...

Adding system user `elasticsearch' (UID 116) ...

Adding new user `elasticsearch' (UID 116) with group `elasticsearch' ...

Not creating home directory `/usr/share/elasticsearch'.

### NOT starting elasticsearch by default on bootup, please execute

sudo update-rc.d elasticsearch defaults 95 10

### In order to start elasticsearch, execute

sudo /etc/init.d/elasticsearch start

Processing triggers for ureadahead ...

$ sudo /etc/init.d/elasticsearch start

* Starting Elasticsearch Server [ OK ]

$

Interact with a web API.

$ curl http://localhost:9200

{

"status" : 200,

"name" : "NFL Superpro",

"version" : {

"number" : "1.1.1",

"build_hash" : "f1585f096d3f3985e73456debdc1a0745f512bbc",

"build_timestamp" : "2014-04-16T14:27:12Z",

"build_snapshot" : false,

"lucene_version" : "4.7"

},

"tagline" : "You Know, for Search"

}

$ curl http://localhost:9200/_cluster/health?pretty

{

"cluster_name" : "elasticsearch",

"status" : "green",

"timed_out" : false,

"number_of_nodes" : 1,

"number_of_data_nodes" : 1,

"active_primary_shards" : 0,

"active_shards" : 0,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 0

}

$

Use Vim to Edit Files over the Network

$ vim scp://remote-host//path/to/file

$ vim scp://remote-user@remote-host//path/to/file

If you want to edit a file with vim over SSH, you can let it do the heavy lifting of copying the file back and forth.

$ vim scp://linuxserver//home/jason/notes.txt