Convert between Unix and Windows text files – dos2unix and unix2dos

 Computers and Networking
Jan 092014
 

How do I convert between Unix and Windows text files?

The format of Windows and Unix text files differs slightly. In Windows, lines end with both the line feed and carriage return ASCII characters, but Unix uses only a line feed. As a consequence, some Windows applications will not show the line breaks in Unix-format files. Likewise, Unix programs may display the carriage returns in Windows text files with Ctrl-m ( ^M ) characters at the end of each line.

There are many ways to solve this problem. This document provides instructions for using FTP, screen capture, unix2dos and dos2unix, tr, awk, Perl, and vi to do the conversion. To use these utilities, the files you are converting must be on a Unix computer.

Note: In the instructions below, replace unixfile.txt with the name of your Unix file, and replace winfile.txt with the Windows filename..

FTP
When using an FTP program to move a text file between Unix and Windows, be sure the file is transferred in ASCII format, so the document is transformed into a text format appropriate for the host. Some FTP programs, especially graphical applications (e.g., Hummingbird FTP), do this automatically. If you are using command line FTP, before you begin the transfer, enter:

ascii
Note: You need to use a client that supports secure FTP to transfer files to and from Indiana University’s central systems. For more, see At IU, what SSH/SFTP clients are supported and where can I get them?

dos2unix and unix2dos
The utilities dos2unix and unix2dos are available for converting files from the Unix command line.

To convert a Windows file to a Unix file, enter:

dos2unix winfile.txt unixfile.txt

To convert a Unix file to Windows, enter:

unix2dos unixfile.txt winfile.txt

tr
You can use tr to remove all carriage returns and Ctrl-z ( ^Z ) characters from a Windows file:

tr -d '\15\32' < winfile.txt > unixfile.txt

However, you cannot use tr to convert a document from Unix format to Windows.

awk
To use awk to convert a Windows file to Unix, enter:

awk '{ sub("\r$", ""); print }' winfile.txt > unixfile.txt

To convert a Unix file to Windows, enter:

awk 'sub("$", "\r")' unixfile.txt > winfile.txt

Older versions of awk do not include the sub function. In such cases, use the same command, but replace awk with gawk or nawk.

Perl
To convert a Windows text file to a Unix text file using Perl, enter:

perl -p -e 's/\r$//' < winfile.txt > unixfile.txt

To convert from a Unix text file to a Windows text file, enter:

perl -p -e 's/\n/\r\n/' < unixfile.txt > winfile.txt

You must use single quotation marks in either command line. This prevents your shell from trying to evaluate anything inside.

vi
In vi, you can remove carriage return ( ^M ) characters with the following command:

:1,$s/^M//g

Note: To input the ^M character, press Ctrl-v , and then press Enter or return.
In command mode in the vi editor I have also used Ctrl-v to get the “^” and then Ctrl-M to get the “M”. Example:
:%s/^M//g inside the vi editor to remove the ^M characters.

The same works for ^[ inside the vi editor. Ctrl-V and Ctrl-[ to get “^[” next to
each other. Example find and replace: :%s/^[//g will remove the ^[ characters from
your document in the vi editor.

In vim, use :set ff=unix to convert to Unix; use :set ff=dos to convert to Windows.

Recursive conversion of files

To recursively convert text files in a directory tree, use dos2unix in combination
with the ‘find’ and ‘xargs’ commands. 

For instance to convert all .txt files under 
the current directory type:

find . -name *.txt |xargs dos2unix

Source for most of this document.

VI Editor Commands

 Computers and Networking
Aug 182012
  
General Startup
	To use vi: vi filename
	To exit vi and save changes: ZZ   or  :wq
	To exit vi without saving changes: :q!
	To enter vi command mode: [esc]

Counts
        A number preceding any vi command tells vi to repeat
	that command that many times.

Cursor Movement

	h       move left (backspace)

	j       move down

	k       move up

	l       move right (spacebar)

	[return]   move to the beginning of the next line

	$       last column on the current line

	0       move cursor to the first column on the
		current line

	^       move cursor to first nonblank column on the
		current line

	w       move to the beginning of the next word or
		punctuation mark

	W       move past the next space

	b       move to the beginning of the previous word
		or punctuation mark

	B       move to the beginning of the previous word,
		ignores punctuation

        e       end of next word or punctuation mark

        E       end of next word, ignoring punctuation

        H       move cursor to the top of the screen 

        M       move cursor to the middle of the screen

        L       move cursor to the bottom of the screen 

Screen Movement

       G        move to the last line in the file

       xG       move to line x

       z+       move current line to top of screen

       z        move current line to the middle of screen

       z-       move current line to the bottom of screen

       ^F       move forward one screen

       ^B       move backward one line

       ^D       move forward one half screen

       ^U       move backward one half screen

       ^R       redraw screen
		( does not work with VT100 type terminals )

       ^L       redraw screen
		( does not work with Televideo terminals )

Inserting

       r        replace character under cursor with next
		character typed

       R        keep replacing character until [esc] is hit

       i        insert before cursor

       a        append after cursor

       A        append at end of line

       O        open line above cursor and enter append mode

Deleting

	x       delete character under cursor

	dd      delete line under cursor

        dw      delete word under cursor

        db      delete word before cursor

Copying Code

        yy      (yank)'copies' line which may then be put by
		the p(put) command. Precede with a count for
		multiple lines.

        :t.     will duplicate the line.

        :t 7    will copy it after line 7.

        :,+t0   will copy current and next line at the
        beginning of the file.

        :1,t$   will copy lines from beginning of the file
        to the current cursor position, to the end of the file.

Put Command
        brings back previous deletion or yank of lines,
	words, or characters

        P       bring back before cursor

        p       bring back after cursor

Find Commands

	?       finds a word going backwards

	/       finds a word going forwards

        f       finds a character on the line under the
		cursor going forward

        F       finds a character on the line under the
		cursor going backwards

        t       find a character on the current line going
		forward and stop one character before it

	T       find a character on the current line going
		backward and stop one character before it

	;	repeat last f, F, t, T

Find and Replace Commands

        :%s/hello/goodbye/g   find hello and replace with goodbye

        :%s/^\(.*\)\n\1$/g    delete duplicate lines

        :%s/\n/ /g            join lines together

Miscellaneous Commands

	.	repeat last command

	u	undo last command issued

	U	undoes all commands on one line

	xp	deletes first character and inserts after
		second (swap)

	J	join current line with the next line

	^G	display current line number

	%	if at one parenthesis, will jump to its mate

	mx	mark current line with character x

	'x	find line marked with character x

	NOTE: Marks are internal and not written to the file.

Line Editor Mode
	Any commands from the line editor ex can be issued
	upon entering line mode.

	To enter: type ':'

	To exit: press[return] or [esc]

ex Commands
	For a complete list consult the
	UNIX Programmer's Manual

READING FILES
	copies (reads) filename after cursor in file
	currently editing

	:r filename

WRITE FILE
	:w 	saves the current file without quitting
	:20,40w filename write the contents of the lines numbered 20 through 40 to
	a new file named filename
MOVING

	:#	move to line #

	:$	move to last line of file

SHELL ESCAPE
	executes 'cmd' as a shell command.

	:!'cmd'

Source:

Text File Format Compatibility (Unix, Linux, Mac, Windows, DOS)

 Computers and Networking
Jul 202012
 

As simple and straightforward as they may seem, text files still harbor an opportunity for compatibility problems. Different operating systems have traditionally used different ways to indicate line endings (line breaks). Mac OS has traditionally used the Carriage Return character (ASCII chcracter 13, aka CR or ^M) to indicate line breaks; unix has traditionally used the Line Feed character (ASCII 10, aka LF or ^J). Since Mac OS X derives from both heritages, it winds up using a mix of the two in various contexts. But most command line utilites only understand (and produce) files with unix-style breaks.Just to make things even more fun, there’s actually a third variant: MS-DOS its successors use a carriage return followed by a line feed to indicate a line break. Few Macintosh programs will generate such files, but if you need to deal with a file that came from a PC, you’ll probably want to convert it to a more native format on the Mac.

Fortunately, it’s fairly easy to convert the formats back and forth on the command line. Here are some examples of how to transform files back and forth:

tr '\r' '\n' <macfile.txt >unixfile.txt
convert the Mac-format file macfile.txt to unix format, and save
the result as unixfile.txt. tr is a program that does character
substitution, and in this case it's simply being used to replace
CR (written \r on the command line) with LF (written \n)
throughout the file.

tr '\r' '\n' <macfile.txt | grep fnord
convert the Mac-format file macfile.txt to unix format, then use
grep to search the file for the word "fnord". (Note: grep doesn't
understand Mac-style line breaks.)

tr '\n' '\r' <unixfile.txt >macfile.txt
convert the unix-format file unixfile.txt to Mac format, and save
the result as macfile.txt.

perl -p -e 's/\r/\n/g' macfile.txt >unixfile.txt
convert the Mac-format file macfile.txt to unix format, and save
the result as unixfile.txt. This is functionally identical to the
first example, but since perl is actually a very general
programming language, it can also do some other useful things...
BTW, he -e means the program will be the next thing on the command
line ('s/\r/\n/g' - perlese for replace all \r's with \n's), and
the -p means do this for each line of the file.

perl -pi -e 's/\r/\n/g' textfile.txt
convert the file textfile.txt from Mac-style (CR) line breaks to
unix-style (LF), and replace the original file with the converted
version (that's what the -i means).

perl -pi -e 's/\r\n?/\n/g' textfile.txt
convert the file textfile.txt from Mac-style (CR) or PC-style
(CRLF) line breaks to unix-style (LF), and replace the original
file.

perl -pi -e 's/\r\n?/\n/g' *.txt
convert all text files (or rather, files with .txt extensions)
in the current directory to unix-style breaks. Note that any that
were already in unix format will not be changed.

perl -pi -e 's/\n/\r/g' textfile.txt
convert the file textfile.txt from unix-style (LF) line breaks to
Mac-style (CR), and replace the original file. Source

Source

Shell script for checking disk space

 Computers and Networking
Sep 072010
 

Filename: disktest.sh

#!/bin/bash
# This script does a very simple test for checking disk space.

space=`df -h | awk ‘{print $5}’ | grep % | grep -v Use | sort -n | tail -1 | cut -d “%” -f1 -`

case $space in
[1-6]*)
  Message=”All is quiet.”
  ;;
[7-8]*)
  Message=”Start thinking about cleaning out some stuff.  There’s a partition that is $space % full.”
  ;;
9[1-8])
  Message=”Better hurry with that new disk…  One partition is $space % full.”
  ;;
99)
  Message=”I’m drowning here!  There’s a partition at $space %!”
  ;;
*)
  Message=”I seem to be running with an nonexistent amount of disk space…”
  ;;
esac

Shell script using case statement to filter user input

 Computers and Networking
Sep 072010
 

It’s possible to use even more complicated syntax with regular expressions:

I used this in my FTP script for safety surveillor.

case “$1? in
+(start|run) ) /usr/app/startup-script ;;
@([Ss])top ) /usr/app/stop-script ;;
esac

?(pattern1 | pattern2 | … | patternn)
zero or one occurrence of any pattern

*( pattern1 | pattern2 | … | patternn)
zero or more occurrences of any pattern

@( pattern1 | pattern2 | … | patternn)
exactly one occurrence of any pattern

+( pattern1 | pattern2 | … | patternn)
one or more occurrence of any pattern

!( pattern1 | pattern2 | … | patternn)
all strings except those that match any pattern

SFTP Commands

 Computers and Networking
Jul 062010
 

On Linux or Unix Systems you can use sftp like this:

sftp -oPort=1234 username@remote.servername.com
After authenticating, the following commands can be used. Any
paths should have quotes if they contain spaces.

Some standard commands and their definitions for command line SFTP include:

?	Get help on the use of SFTP commands.
bye	Close the connection to the remote computer and exit SFTP.
cd	Change the directory on the remote computer.
chgrp	Change the group associated with a computer file (chgrp system foofile).
chmod	Change the permissions of files on the remote computer.
chown	Change the owner of files on the remote computer.
dir	List the files in the current directory on the remote computer.
exit	Close the connection to the remote computer and exit SFTP.
get	Copy a file from the remote computer to the local computer.
help	Get help on the use of SFTP commands.
lcd	Change the directory on the local computer.
lls	See a list of the files in the current directory on the local computer.
lmkdir	Create a directory on the local computer.
ln	Create a symbolic link for a file on the remote computer.
lpwd	Show the current directory (present working directory) on local computer.
ls	Show the current directory on the remote computer.
lumask	Change the local umask value.
mkdir	Create a directory on the remote computer.
progress Toggle display of progress meter.
put	Copy a file from the local computer to the remote computer.
pwd	Show the current directory (present working directory) on remote computer.
quit	Close the connection to the remote computer and exit SFTP.
rename	Rename a file on the remote host.
rm	Delete files from the remote computer.
rmdir	Remove a directory on the remote host (the directory has to be empty).
symlink	Create a symbolic link for a file on the remote computer.
version	Display the SFTP version.
! 	In Unix, exit to the shell prompt, where you can enter commands. Enter exit
	to get back to SFTP. If you follow  !  with a command (e.g., !pwd), SFTP
	will execute the command without dropping you to the Unix prompt.

Tac command in Linux

 Computers and Networking
Jun 242010
 

Before we get started with the commands you may not have heard of before, here’s a tried and true Unix command that we’ve all been using for decades, but maybe not this particular way. This grep command represents the simplest way to remove blank lines from a file.

grep . oldfile > newfile

Got that? It matches any lines with anything (i.e., not nothing) in them and tosses them into the second file. Hey, this is even easier than my old “grep -v ^$ oldfile > newfile” command that would specifically omit empty lines from the new file
Another interesting command is called “tac”. Yes, that’s “cat” spelled backwards. And what would you expect such a command to do? It displays a file in reverse order — upside down. So, our a, b, c, x, a, b, c file shows as:

$ tac myfile
c
b
a
x
c
b
a

Using awk

 Computers and Networking
Jun 242010
 

Awk does a lot more than select a column from a file or an input stream. It can select columns from selected rows. It can calculate totals, extract substrings, reverse the order of fields and provide a whole lot of other very handy manipulations. Whether you squeeze your awk permutations onto the command line or prefer to build them into scripts, the language is clever and versatile and well worth using even as it joins the ranks of middle-aged computer utilities.

If you want to use awk to add up a bunch of numbers formulated as a column in a text file, you can use a one-liner like this:

$ awk ‘{ SUM+=$1 } END { print SUM }’ < nums
That SUM+= operation adds the contents of column one to a running total for each line of input. The sum is only printed at the very end when input is exhausted. You can change $1 to the column of your choice or $NF if you want to add up the rightmost column.

If you prefer scripts to the command line, you could use a script like the addcol script below to sum whatever column you choose. You would just pass the column you want to add on the command line as the COL parameter:

$ awk -f addcol COL=3 < numbers
The -f tells awk to run the designated script (addcol) and COL=3 passes “3” as the column number you want to sum.

# addcol
BEGIN { SUM=0 }
{
  print $COL
  SUM += $COL
}
END { print “Sum: ” SUM }
Awk has some grep-like features as well. If you want to operate only on lines that contain some particular text, you can specify that text on the command line like this:

$ awk ‘/choose me/’ textfile
You can also combine search options using && (and), || (or), and even negation operators. The command below selects lines that contain both the word “this” and the word “that”. Using ‘/this/ && !/that/’ would select only lines that contain “this” without containing “that”.

$ awk ‘/this/ && /that/’ notes
# find the process that started this one
# make sure that this user provided an answer
You can also select content based on its position within your input. In the command below, we only want to see lines eleven and above.

$ awk ‘NR > 10’ counts
11 63
12 99
13 63
14 77
15 41

And, of course, there are a lot of other nice little tricks available.

Awk provides some quick and effective filtering and incorporates an easy syntax. Probably the only thing that takes some getting used to is referring to parameters without putting dollar signs in front of them.

Obtained from

Clear MySQL Command Line History

 Computers and Networking
Apr 152010
 

Q. How do I clear the mysql command line history stored in ~/.mysql_history file?

A. mysql is a simple SQL shell. It supports interactive and non-interactive use.

On Unix and Linux, the mysql client writes a record of executed statements to a history file. By default, the history file is named .mysql_history and is created in your home directory. To specify a different file, set the value of the MYSQL_HISTFILE environment variable.

Task: CLEAR MYSQL COMMAND LINE HISTORY

Remove ~/.mysql_history by typing following command:
$ > ~/.mysql_history

If you do not want to maintain a history file, first remove .mysql_history if it exists, and then use either of the following techniques:

Set the MYSQL_HISTFILE variable to /dev/null. To cause this setting to take effect each time you log in, put the setting in one of your shell’s startup files.

Create ~/.mysql_history as a symbolic link to /dev/null. You need run following command only once:
$ rm $HOME/.mysql_history
$ ln -s /dev/null $HOME/.mysql_history