the empty quarter » commandline

Column-wise text manipulation

martin — Mon, 17 Oct 2011 17:02:14 +0000

GNU/Linux includes many utilities for working with text files through the shell. In this post we take a quick look at accessing and manipulating text files in a “column-wise” mode.

Suppose you have the following two files, each with two columns separated by the TAB character.
$cat file1 Alice Paris Bob Tokyo Mary London John New York

$cat file2 13 May Orange 19 Oct Blue 11 Nov Black 29 Feb Red

The data in the two files are in fact related, i.e. file2 contains the date of birth and favourite colour of the people mentioned in file1 (assuming also that the files are sorted correctly). It would make sense to combine the two files together so that each row has the full data for each person. The paste command does just that.
$paste file1 file2 > file3 $cat file3 Alice Paris 13 May Orange Bob Tokyo 19 Oct Blue Mary London 11 Nov Black John New York 29 Feb Red

Suppose that we are only interested in the name and date of birth of each person, and we can discard the hometown and favourite colour information. The cut command is what we shall use:
$cut file3 -f 1,3 > file4 $cat file4 Alice 13 May Bob 19 Oct Mary 11 Nov John 29 Feb

Our next and final requirement is to reorder the columns differently. Instead of having the name followed by date of birth, suppose we want to have the columns the other way round. Unfortunately cat -f 3,1 produces exactly the same output as cut -f 1,3, so the cut command will not be sufficient. We have to use sed instead.
$sed -e 's/$[^\t]*$\t$[^\t]*$/\2\t\1/' file4 > file5 $cat file5 13 May Alice 19 Oct Bob 11 Nov Mary 29 Feb John

How does that work? Well $[^\t]*$ is a “named expression” which matches all characters except TAB. The search pattern looks for two of them, separated by TAB (\t). In the replace-with part, they are referred to as \2 and \1, again separated by \t.

Of course if file5 was what we ultimately wanted from the beginning as our output, we could have simply piped commands together:
$paste file1 file2 | cut -f 1,3 | sed -e 's/$[^\t]*$\t$[^\t]*$/\2\t\1/' > file5

or alternatively
$paste file1 file2 | sed -e 's/$[^\t]*$\t$[^\t]*$\t$[^\t]*$\t$[^\t]*$/\3\t\1/' > file5

Remove HTML tags with sed

martin — Mon, 17 Oct 2011 12:16:13 +0000

Sed can be used to strip out all HTML or XML tags from a file and get the plain text version. Suppose you have file gnulinux.html with the following contents:

The combination of GNU and Linux is the GNU/Linux operating system, now used by millions and sometimes incorrectly called simply “Linux“.

Tempting but incorrect – sed finds the longest possible match which in this case is the entire file, and thus will output nothing:
$sed -e 's/<.*>//g' gnulinux.html

Correct version:
$sed -e 's/<[^>]*>//g' gnulinux.html The combination of GNU and Linux is the GNU/Linux operating system, now used by millions and sometimes incorrectly called simply “Linux“.

Remove empty lines from file

martin — Wed, 22 Sep 2010 16:50:07 +0000

Here’s a quick way to remove empty lines from a file using the Linux command line:

cat file1 | sed /^$/d > file2

Where file1 is the input file containing empty lines and file2 is a newly created file with empty lines removed.

How to change SSH port number on Ubuntu server

martin — Fri, 03 Sep 2010 08:56:48 +0000

Changing the port number of SSH daemon is a quick way of reducing the number of SSH brute force attacks your server might face (check the file /var/log/auth.log to see if there are many failed SSH login attempts).

Just to be on the safe side, create a backup copy of the SSH daemon config file.
sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.vanilla
Edit the config file.
sudo vi /etc/ssh/sshd_config
Change the port number on the following line, e.g. to 2201 or some other unused port. Make sure you note down the port number.
Port 22
Restart the SSH daemon. You might get kicked out of your existing session.
sudo /etc/init.d/ssh restart
When you login next remember to include the correct port.
ssh youruser@yourserver -p 2201

Tuxradar: Command line tricks for smart geeks

martin — Wed, 16 Dec 2009 13:46:40 +0000

Good article on Tuxradar with some clever command line hacks: Command line tricks for smart geeks

Use commandline to find CentOS version

martin — Wed, 18 Nov 2009 17:58:42 +0000

Here’s how to find the version of your CentOS system from the command line:

more /etc/redhat-release

Change virtual terminal font size

martin — Mon, 21 Sep 2009 05:28:14 +0000

This post is about how to change the resolution (i.e. font size) of virtual terminals or “tty” in Linux. Virtual terminal is the basic command line interface which you see when you press (note: to get back to your existing Gnome or KDE session press ). This is different from the Terminal emulator, such as the Terminal application in Gnome.

To change the font size of the virtual terminal it is necessary to set the vga parameter in the entry used by the Grub bootloader to start up Linux. Follow the steps below to set the entry correctly.

First, intall the hwinfo utility through Synaptic or Terminal:
sudo apt-get install hwinfo
Then run the following command in Terminal:
sudo hwinfo --framebuffer
The command will output some information about your graphics system. Skip the top bit and you should see a list of entries like this:
Mode 0x0300: 640x400 (+640), 8 bits Mode 0x0301: 640x480 (+640), 8 bits Mode 0x0303: 800x600 (+800), 8 bits Mode 0x0305: 1024x768 (+1024), 8 bits Mode 0x0307: 1280x1024 (+1280), 8 bits Mode 0x0317: 1024x768 (+2048), 16 bits Mode 0x0318: 1024x768 (+4096), 24 bits
Review the list which might be quite long, select the resolution you want to use and copy the hexadecimal number after Mode. For example, for 24 bit 1024×786 resolution the number is 0×0318.
Still in the Terminal emulator, run the following command to edit the Grub bootloader configuration file:
gksudo gedit /boot/grub/menu.lst
Find the entry you want to edit (or copy an existing entry and give it a different name) and change the line which begins with “kernel”. As a precaution, make sure you have at least one entry left in the file without the new resolution so you can still safely boot and edit the file again if necessary.

Original entry:
kernel /boot/vmlinuz-2.6.28-15-server root=/dev/sda3 ro quiet splash

New entry:
kernel /boot/vmlinuz-2.6.28-15-server root=/dev/sda3 ro quiet vga=0x0318 splash
Reboot your computer and at the Grub menu select the entry which was edited. You may notice some difference during the boot process, to verify that the resolution and font size are suitable press once you are at the login screen. This will take you to tty8 which should display messages from the boot process.

Split large files

martin — Fri, 18 Sep 2009 14:33:20 +0000

Sometimes it is necessary to split a large file into several smaller ones, for example so that they fit on a CD or USB disk or can be attached to email. It’s very easy to do this from command line.

Suppose you have a file of 500 MiB called “large_file” and to split it into 5 files of 100 MiB each:

split --bytes=100m large_file large_file_part_

This will create 5 files with names as follows:

large_file_part_aa large_file_part_ab large_file_part_ac large_file_part_ad large_file_part_ae

The following command would then be used to join the parts back together:

cat large_file_part_* > large_file

Mount NTFS disk from commandline

martin — Sun, 07 Jun 2009 15:56:09 +0000

Nautilus and Thunar sometimes refuse to mount NTFS disk (e.g. a USB hard drive) when it was previously not unmounted cleanly from Windows. It is still possible to mount it from terminal shell by using the following command:

sudo mount -t ntfs-3g -o force /dev/sdc1 /media/myntfsdisk

Where /dev/sdc1 is the device you want to mount and /media/myntfsdisk is the mount point.

Check installed packages from commandline

martin — Thu, 28 May 2009 13:31:24 +0000

To check for installed packages from the command line, use the dpkg command with the --get-selections option. This returns all packages installed on your system (most likely a lot!). So it is best to use grep to narrow down the search results. For example, the following command lists all packages with “fire” in the package name:

dpkg --get-selections | grep fire firefox install firefox-3.0 install firefox-3.0-branding install firefox-3.0-dom-inspector install firefox-dom-inspector install mozilla-firefox-locale-en-gb install

An alternative which shows more information, including the exact version of the package and a description is the -l option. Because grep works with the output of the dpkg command, this will also include packages where the search term appears in the description, such as ufw on the last line in the output below because of “firewall”.
$ dpkg -l | grep fire ii firefox 3.0.10+nobinonly-0ubuntu0.9.04.1 meta package for the popular mozilla web bro ii firefox-3.0 3.0.10+nobinonly-0ubuntu0.9.04.1 safe and easy web browser from Mozilla ii firefox-3.0-branding 3.0.10+nobinonly-0ubuntu0.9.04.1 Package that ships the firefox branding ii firefox-3.0-dom-inspector 3.0.10+nobinonly-0ubuntu0.9.04.1 dummy upgrade package for firefox-3.0-dom-in ii firefox-dom-inspector 3.0.10+nobinonly-0ubuntu0.9.04.1 meta package for firefox-dom-inspector ii mozilla-firefox-locale-en-gb 2.0.0.7+1-0ubuntu4 Mozilla Firefox English language/region pack ii ufw 0.27-0ubuntu2 program for managing a netfilter firewall

Using the above commands can be a faster and more efficient alternative to searching for installed packages in Synaptic.