Files and Directories - Command Line Kung Fu (2014)

Command Line Kung Fu (2014)

Files and Directories

Quickly Make a Backup of a File

$ cp file{,.bak}

You can use brace expansion to quickly create a backup copy of a file. Brace expansion allows you to create multiple command line arguments from a single argument. The single argument is combined with all the strings that are given within the braces and creates as many new arguments as brace strings. Values in the braces can either be comma separated strings or a sequence expression. Examples of sequence expressions include {1..5} and {a..z}.

$ sudo cp /etc/passwd{,.bak}

$ ls /etc/passwd*

/etc/passwd /etc/passwd.bak

$ mkdir -p ~/my-app/{bin,lib,log}

$ ls ~/my-app/

bin lib log

$ echo 10.0.0.{0..7}

10.0.0.0 10.0.0.1 10.0.0.2 10.0.0.3 10.0.0.4 10.0.0.5 10.0.0.6 10.0.0.7

Quickly Change a File's Extension

$ mv file{.old,.new}

To rename a file with a new extension employ brace expansion. This example changes a ".txt" file to a ".doc" file.

$ ls report*

report.txt

$ mv report.{txt,doc}

$ ls report*

report.doc

$

Here's another example.

$ ls httpd*

httpd.cfg

$ mv httpd.{cfg,conf}

$ ls httpd*

httpd.conf

$

This command will add an extension to a file.

$ ls jazz*

jazz

$ mv jazz{,.mp3}

$ ls jazz*

jazz.mp3

$

Create Backups of Files by Date with Ease

$ alias d='date +%F'

By using the date format of YYYY-MM-DD for file or directory names, you make ls output more human friendly. You can quickly tell the date order of the files when using this format.

$ ls -1 file*

file.2013-04-08

file.2013-12-21

file.2014-04-08

$ ls -1 other-file.*

other-file.04-08-2013

other-file.04-08-2014

other-file.12-21-2013

Before making a change to a file I like to make a backup copy of it. I also like to know when I made the backup. By creating an alias of "d" which is short for the date in YYYY-MM-DD format, I can quickly create these types of backups.

$ echo "alias d='date +%F'" >> ~/.bash_profile

$ ~/.bash_profile

$ d

2014-04-08

$ sudo cp httpd.conf httpd.conf.`d`

$ sudo cp httpd.conf httpd.conf.$(d)

$ sudo cp httpd.conf !#.$(d)

$ sudo cp httpd.conf{,.$(d)}

$ ls -1 httpd.conf*

httpd.conf

httpd.conf.2014-04-08

Overwrite the Contents of a File

$ command > file

You can redirect output from one command to a file using the greater-than symbol. A single greater-than symbol (>) redirects standard output to a file, overwriting (truncating) any existing contents of the file. If no file exists, it creates one.

$ grep bash /etc/passwd > users-that-use-bash

$ cat users-that-use-bash

root:x:0:0:root:/root:/bin/bash

jason:x:501:501:Jason:/home/jason:/bin/bash

oracle:x:1006:1006::/home/oracle:/bin/bash

bob:x:1000:1000:Bob Smith:/home/bob:/bin/bash

$

Empty a File That Is Being Written To

$ > file

$ cat /dev/null > file

To quickly zero out a file you can redirect nothing to it. Why not just delete the file and recreate it? If a process has a file open when you delete it, the process will keep the file handle open and continue writing to that file. Only when the process closes the file handle will the disk space used by that file be freed. If an application fills up /var and you delete the open log file, /var will still be full. If you redirect nothing to the file, the file is truncated and the application can continue writing to the file.

$ sudo -s

# > /var/log/maillog

# ls -l maillog

-rw-------. 1 root root 0 Apr 9 18:55 maillog

Append a String to a File

$ command >> file

The double greater than sign (>>) redirects standard output to a file and appends to any existing contents. If no file exists, it creates one.

$ echo build new server >> todo.txt

$ cat todo.txt

build new server

$ echo add server to the load balancer >> todo.txt

$ cat todo.txt

build new server

add server to the load balancer

$

Follow a File as It Grows

$ tail -f file

For a file that is constantly being update like a log file, use tail -f to view the updates to the file in real time.

$ tail -f /var/log/syslog

Apr 10 21:45:01 linwww1 CRON[31769]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

Apr 10 21:46:28 linwww1 whoopsie[1421]: online

Apr 10 21:47:37 whoopsie[1421]: last message repeated 2 times

Apr 10 21:55:01 linwww1 CRON[32548]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

Apr 10 22:05:01 linwww1 CRON[931]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

Apr 10 22:15:01 linwww1 CRON[2459]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

Apr 10 22:17:01 linwww1 CRON[2609]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)

Apr 10 22:25:01 linwww1 CRON[3197]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

Apr 10 22:35:01 linwww1 CRON[4036]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

Apr 10 22:35:33 linwww1 whoopsie[1421]: online

Watch Multiple Log Files at the Same Time

$ multitail file1 fileN
$ multitail file1 -I fileN

The multitail command allows you to browse through several files at once. Not only does multitail allow you to watch multiple files, it supports color highlighting, filtering, merging, and more. Here is a quick rundown of some of the most helpfuls commands for multitail.

F1 - Help

a - Add another file to follow

d - Delete a file from the view

/ - Start a search (Find)

ctrl-g - Exit a command, menu, or action. Similar to the emacs ctrl-g key binding.

q - Quit

$ sudo multitail /var/log/syslog /var/log/kern.log

Apr 19 12:43:29 linuxsvr NetworkManager[758]: <info> (eth0): IP6 addrconf timed out or failed.

Apr 19 12:43:29 linuxsvr NetworkManager[758]: <info> Activation (eth0) Stage 4 of 5 (IPv6 Configure Timeout) scheduled...

Apr 19 12:43:29 linuxsvr NetworkManager[758]: <info> Activation (eth0) Stage 4 of 5 (IPv6 Configure Timeout) started...

Apr 19 12:43:29 linuxsvr NetworkManager[758]: <info> Activation (eth0) Stage 4 of 5 (IPv6 Configure Timeout) complete.

00] /var/log/syslog *Press F1/<CTRL>+<h> for help* 132KB - 2014/04/19 12:46:45

Apr 19 12:43:08 linuxsvr kernel: [ 10.012891] Console: switching to colour frame buffer device 240x67

Apr 19 12:43:08 linuxsvr kernel: [ 10.012926] fb0: VESA VGA frame buffer device

Apr 19 12:43:11 linuxsvr kernel: [ 12.288420] hda-intel: Invalid position buffer, using LPIB read method instead.

Apr 19 12:43:17 linuxsvr kernel: [ 18.891011] hda-intel: IRQ timing workaround is activated for card #0. Suggest a bigger bdl_pos_adj.

01] /var/log/kern.log *Press F1/<CTRL>+<h> for help* 132KB - 2014/04/19 12:46:45

To merge multiple files into one window, use the "-I" option. This intermixes the output of both of the files. This can aid in troubleshooting a problem.

$ sudo multitail /var/log/syslog -I /var/log/kern.log

Apr 19 12:43:08 linuxsvr kernel: [ 10.012891] Console: switching to colour frame buffer device 240x67

Apr 19 12:43:08 linuxsvr kernel: [ 10.012926] fb0: VESA VGA frame buffer device

Apr 19 12:43:11 linuxsvr kernel: [ 12.288420] hda-intel: Invalid position buffer, using LPIB read method instead.

Apr 19 12:43:17 linuxsvr kernel: [ 18.891011] hda-intel: IRQ timing workaround is activated for card #0. Suggest a bigger bdl_pos_adj.

Apr 19 12:43:29 linuxsvr NetworkManager[758]: <info> Activation (eth0) Stage 4 of 5 (IPv6 Configure Timeout) complete.

Apr 19 12:43:29 linuxsvr NetworkManager[758]: <info> Activation (eth0) Stage 4 of 5 (IPv6 Configure Timeout) scheduled...

Apr 19 12:43:29 linuxsvr NetworkManager[758]: <info> Activation (eth0) Stage 4 of 5 (IPv6 Configure Timeout) started...

Apr 19 12:43:29 linuxsvr NetworkManager[758]: <info> (eth0): IP6 addrconf timed out or failed.

00] /var/log/syslog *Press F1/<CTRL>+<h> for help* 132KB - 2014/04/19 12:48:32

Delete Empty Directories

$ find . -type d -empty -delete

If you ever find yourself needing to clean up empty directories, find can make that task a snap. Use the -type d option to find all the directories with the -empty option to only include empty directories and finally the -delete option removes the directories.

$ mkdir -p testing/{1..4}

$ ls testing/

1 2 3 4

$ find . -type d -empty -delete

$ ls testing/

ls: cannot access testing/: No such file or directory

$

Print a List of Files That Contain a given String

$ grep -rl string .

To get a list of files that contain a given string, use grep with the -r (recursive) and -l (list files that match) options.

$ sudo grep -lr jim /var/log

/var/log/audit/audit.log.1

/var/log/secure-20140406

/var/log/secure

$

An Easy-to-Read Recursive File Listing

$ find . -type f -ls

The ls command has a recursive option, but I find reading the output from the find command to be easier to digest, especially for a large number of files and directories. The advantage to using find is that it displays the full path to each file, unlike ls.

$ ls -lR Music/

Music/:

total 4

drwxr-xr-x. 2 jason jason 4096 Feb 22 12:40 jazz

Music/jazz:

total 20932

-rw-r--r--. 1 jason jason 79496 Feb 22 12:40 giant-steps.mp3

-rw-r--r--. 1 jason jason 21267371 Feb 16 11:12 jazz-album-1.mp3

-rw-r--r--. 1 jason jason 79496 Feb 3 18:13 john-coletrane.mp3

$ find Music/ -type f -ls

397966 20772 -rw-r--r-- 1 jason jason 21267371 Feb 16 11:12 Music/jazz/jazz-album-1.mp3

396787 80 -rw-r--r-- 1 jason jason 79496 Feb 22 12:40 Music/jazz/giant-steps.mp3

132464 80 -rw-r--r-- 1 jason jason 79496 Feb 3 18:13 Music/jazz/john-coletrane.mp3

$

View Files and Directories in a Tree Format

$ tree

$ tree -d

$ tree -L number

The tree command displays files and directories in a tree like format. If you only want to see the directory structure, use the -d option. To limit the depth of the tree, use the -L option followed by a number.

$ tree Music/

Music/

└── jazz

├── giant-steps.mp3

├── jazz-album-1.mp3

└── john-coletrane.mp3

1 directory, 3 files

$ tree -d Music/

Music/

└── jazz

1 directory

$ tree -d /var | head

/var

├── account

├── cache

│ ├── fontconfig

│ ├── gdm

│ │ ├── adminuser

│ │ └── jason

│ ├── hald

│ ├── ldconfig

│ ├── man

$ tree -dL 1 /var

/var

├── account

├── cache

├── crash

├── db

├── empty

├── ftp

├── games

├── gdm

├── lib

├── local

├── lock

├── log

├── mail -> spool/mail

├── nis

├── opt

├── preserve

├── run

├── spool

├── tmp

├── www

└── yp

$

Replace a String in Multiple Files

$ find /path -type f -exec sed -i.bak 's/string/replacement/g' {} \;

If you have some information that is embedded in multiple files and you need to change it, use the find command in combination with sed. The -i option for sed tells it to perform an in-place edit. You can supply an extension to -i to create a backup of the file before the edits are made. The command passed to sed tells it to substitute "replacement" for "string" globally. A global substitution will perform the replacement everywhere. A normal substitution replaces only the first occurrence on a line.

In this example the server that houses software repositories is being replaced. With the following one-liner, all occurrences of "thor" are replaced by "loki."

$ grep -r thor .

./deploy-il.sh:REPO_SERVER="thor.company.com"

./deploy-ca.sh:REPO_SERVER="thor.company.com"

./deploy-fl.sh:REPO_SERVER="thor.company.com"

./deploy-ny.sh:REPO_SERVER="thor.company.com"

$ find . -type f -exec sed -i.bak 's/thor/loki/g' {} \;

$ grep -r thor .

./deploy-fl.sh.bak:REPO_SERVER="thor.company.com"

./deploy-il.sh.bak:REPO_SERVER="thor.company.com"

./deploy-ny.sh.bak:REPO_SERVER="thor.company.com"

./deploy-ca.sh.bak:REPO_SERVER="thor.company.com"

$ grep -r loki .

./deploy-il.sh:REPO_SERVER="loki.company.com"

./deploy-ca.sh:REPO_SERVER="loki.company.com"

./deploy-fl.sh:REPO_SERVER="loki.company.com"

./deploy-ny.sh:REPO_SERVER="loki.company.com"

$

Extract the Nth Line from a File

$ awk 'NR==N'

To extract a specific line from a file, tell awk to print the line that matches the line number. One area where this comes in helpful is when you encounter an error in a script. To see exactly what the script is doing on that line, use awk.

$ ./deploy.sh

./deploy.sh: line 74: /usr/local/bin/patch: No such file or directory

$ awk 'NR==74' deploy.sh

/usr/local/bin/patch $NEW_HOST

$

Convert Text Files from Windows Format to Linux Format and Vice-Versa

$ dos2unix

$ unix2dos

Sooner or later you're going to be sent a file or download one that uses a pair of CR (carriage return) and LF (line feed) characters to terminate lines in the file. Those type of files are Windows/DOS formatted. Unix-like operating systems simply use the LF character to terminate a line. Sometimes this can cause issues. To convert the file to a unix-like format, use dos2unix. To examine the line termination characters use "cat -A" or the "file" command.

$ cat -A from-my-windows-buddy.pl

#!/usr/bin/perl^M$

print "This is a cross-platform perl script!\n"^M$

$ ./from-my-windows-buddy.pl

-bash: ./from-my-windows-buddy.pl: /usr/bin/perl^M: bad interpreter: No such file or directory

$ dos2unix from-my-windows-buddy.pl

dos2unix: converting file from-my-windows-buddy.pl to UNIX format ...

$ cat -A from-my-windows-buddy.pl

#!/usr/bin/perl$

print "This is a cross-platform perl script!\n"$

$ ./from-my-windows-buddy.pl

This is a cross-platform perl script!

$ file other-file.txt

other-file.txt: ASCII text, with CRLF line terminators

$ dos2unix other-file.txt

dos2unix: converting file other-file.txt to UNIX format ...

$ file other-file.txt

other-file.txt: ASCII text

$

The opposite side of this coin is that if you send a file created on a Linux host to someone who opens it in Notepad on Windows, they will see just one long line of text. Convert the file to Windows/DOS format with the unix2dos command.

$ file report-for-ceo.txt

report-for-ceo.txt: ASCII text

$ unix2dos report-for-ceo.txt

unix2dos: converting file report-for-ceo.txt to DOS format ...

$ file report-for-ceo.txt

report-for-ceo.txt: ASCII text, with CRLF line terminators

$