Showing posts with label Unix. Show all posts
Showing posts with label Unix. Show all posts

Thursday, 10 December 2015

12/10/2015 02:44:00 am

Control M Character in File --Everything explained

This is the problem which every IT Person faces when dealing with files across platforms.If you haven't faced it I can guarantee you will face it one day.



There is lot of information on internet about this but it has not been clearly explained.I will try to put all information together about Control M character at one place.

First of all it is important to understand

"Line Endings" in different operating systems 

Line Ending means when you are at end of line and how Enter Key is interpreted in different operating systems.

All major operating system interpret Enter character differently

OS Line Ending
Windows CR/LF
Unix LF
Mac CR/LF

CR = \r = Carriage Return
LF = \n = Line Feed
Why Do Control M Character Occur

When we  open a text file that was created under Windows. The text characters in the first line are all displayed correctly. At the end of the line, we find a CR. This means nothing special to Unix so the unix attempts to display the character. CR is a non-printable character. Under Unix many of these non-printable characters are mapped to control characters when displayed. In the case of CR, its displayable equivalent is Control-M. This is displayed in most editors as ^M.
The next character is a LF. Unix is fine with it as it is the standard line end character. The editor therefore moves to the start of the next line on the display and starts to process the next line.

How to view Control M Character in Unix

Below will show all  tabs, vertical tabs, carriage returns, linefeeds and whatnot using the slash notation.
od -c Yourfile.txt

If you only want to see control M Character

cat -v filename.txt
Possible Solutions

  1. Remove character in VI Editor Open the VI Editor and do :1,$/^M//g .^M character can be produced by  holding down the control key whilst pressing M at the same time
  2. Different Unix Command : There are various commands availaible to remove control m character    1)  tr -d '\r' < infile.txt > outfile.txt 2) dos2unix < DOSfile.txt > Unixfile.txt
  3. Fix during FTP : Control M character will not appears if files is transferred using ASCII Mode
If you need to process such file in informatica you can call above commands in pre session to remove control M character

 I Hope you are clear about why Control M character occur and possible solutions




1
By
at 1 comment:
Labels: ,

Thursday, 12 November 2015

11/12/2015 01:53:00 pm

All about sftp

SFTP, (SSH File Transfer Protocol) is a protocol packaged with SSH that works in a similar way over a secure connection. The advantage is the ability to leverage a secure connection to transfer files and traverse the file system on both the local and remote system.

 sftp performs all operations over an encrypted ssh session. It uses many of the features of ssh, such as public key authentication and data compression.


sftp can be primarily used in 4 ways 

Interactive session :In this mode, sftp connects and logs into the specified host, then enters its interactive command mode, where you type all your commands at a prompt. To launch an interactive session of sftp, use the following syntax:

 sftp user@server.myhost.com

 Automatic Retrieval Mode : We can use sftp to retrieve files automatically, without any promp

 Batch Mode :  We can write sftp scripts to do ftp in batch mode and these scripts can be called by cron jobs .

 Interactive Mode in Remote Directory : We can start start an interactive mode session in a specific remote directory. We can do this by specifying it on the command line sftp user@server:Dirname






0
By
at No comments:
Labels:

Thursday, 10 April 2014

Sed/Awk Top Commands(Unix interview questions)
4/10/2014 02:14:00 pm

Sed/Awk Top Commands(Unix interview questions)

In this post i will mention very useful sed/awk commands,These are quite handy in day to day work and also useful in unix interview.

How to Print only Blank Line of File.
sed -n '/^$/p' Test_file.txt

To Print First and Last Line using Sed Command

sed  -n ‘1p’ Test_file.txt

sed –n ‘$p’ Test_file.txt

To Print all line Except First Line

sed –n ‘1!p’ Test_file.txt

Delete all Line except First Line

sed –n ‘1!d’ Test_file.txt

How to get only Zero Byte files which are present in the directory

ls -ltr| awk '/^-/ { if($5 ==0) print $9 }'

How add a First record and Last Record to the current file in Linux

sed -i -e '1i Header' -e '$a Trailor' test_file.txt

How to display Even number of records into one file and Odd number of records into another file

awk 'NR %2 == 0' test_files.txt
awk 'NR %2 !=  0' test_files.txt

Remove all empty lines:

sed '/^$/d' test_file.txt

sed '/./!d' test_file.txt

Add at start of line :
awk '{print "START"$0}' FILE 

 Add at end of line :
awk '{print $0"END"}' FILE 

 Too see a particular line

 For example if you just want to see 180th line in file sed -n '180p' testfile.txt

 To find a particular column in file 

 cat testfile.txt |awk -F"," '{print $2}'

 To rename file with current date 

 mv test test_`date +%Y-%m-%d

 Command to take out all those lines which are having 8 at 17th position 
grep '^.\{16\}8' testfile.txt >testfile_new.txt

 To remove nth line without openning file 
sed 'nd' file1>file2 to remove multiple lines sed -e 1d -e 5d

 To find top 20 files with most space

 ls -ltr|awk -F" " '{print $5 $9}' sort -ntail -20

 To find record in first file not in second 
comm -13 testfile1.txt testfile2.txt

 If you are looking from something that is contained in a file but you don't know which directory it is in do the following:
find . -name "*" xargs grep -i something This will find all of the files in the directory and below and grep for the string something in those files!

 Delete Files Delete all the files starting with name testfile 

 find . -type f -name "testfile*" exec rm -f {} \;

 Remove blank space from file 

 sed -e "s/ *//g" testfile.txt >testfile.txt_wo_space

 How to do grep on large number of files in a direcory 

 Normally you get the error "grep argument list too long" if you try to do grep in directory having large number of files ,It can be avoided by using it in below manner

 grep -rl "Search Text" /tmp
How to do count of lines in ebicdic file 

 dd if=ebcdic_file bs=$REC_SIZE > /dev/null

9
By
at 9 comments:
Labels:

Sunday, 22 February 2009

Unix Interview Questions (Part1)
2/22/2009 05:18:00 am

Unix Interview Questions (Part1)

Following are some unix commonly asked interview questions  

Q 1  What is command to check space in Unix 
Ans : df -k 

Q 2 If a file has permission 000 then who can access the File 
Ans : System Administrator .

Q 3 What is command to kill last background Job 
Ans : kill $!

Q 4 How you will list all Hidden files 
Ans : ls -la|grep ^[.] 

Q 5 What is command to create Zero Byte File 
Ans : touch filename 

Q 6  What is difference between diff and cmp command 
Ans : cmp -It compares two files byte by byte and displays first mismatch. 
           diff -It displays all changes required to make files identical.

Q 7  What does $# stands for 
Ans : It will return the number of parameters passed as command line argument.

Q 8 How many prompts are availaible in Unix System
Ans : PS1 the default prompt 
          PS2 Multiline Prompt 

Q 9 How to kill a process forcibily 
Ans : kill -9 PID 
PID (unique identifier of process) 

Q 10 How to create a hidden file 
Ans : While creating file put . DOT in front of file name

If you need more interview questions please mail us at support@itnirvanas.com.

My Profile
2
By
at 2 comments:
Labels:

Monday, 2 February 2009

Unix Script to Automate FTP Process
2/02/2009 08:36:00 am

Unix Script to Automate FTP Process

Unix Script to Automate FTP Process
Normally we come across the situation when we have to FTP file daily.Doing this process daily for large no of files can be cumbersome.We can create Unix script to automate FTP Process

Currently we need to last day files from production to dev box for testing,so we created below mentioned script to FTP files daily

b=`TZ=CST+24 date +%y%m%d` ###To get last date
cd /export/home/mydir
ftp -v -n FTP_HOST_NAME EOF ##Please put two less than sign before EOF
user Userid Pwd
bin
cd /export/home/source_dir
get FILE_NAME.`echo $b`
bye
EOF

Please note replace FTP_HOST_NAME,Userid,Pwd with your actual userid,password and ftp host name.

Please put two less than sign before EOF in third line of script as i am not able to display those characters due to some restrictions.

If you have further queries then please mail to support@itnirvanas.com
1
By
at 1 comment:
Labels:

Friday, 2 January 2009

Unix Handy Commands
1/02/2009 07:35:00 am

Unix Handy Commands

These are the commands which i use on daily basis in Unix.I found these commands very useful

In point 2 and 6 it is not displaying pipe symbol due to some problem.Please use pipe between commands

  1. Too see a particular line For example if you just want to see 180th line in file sed -n '180p' testfile.txt
  2. To find a particular column in file cat testfile.txt awk -F"," '{print $2}'
  3. To rename file with current date mv test test_`date +%Y-%m-%d
  4. This command will take out all those lines which are having 8 at 17th position grep '^.\{16\}8' testfile.txt >testfile_new.txt
  5. To remove nth line without openning file sed 'nd' file1>file2 to remove multiple lines sed -e 1d -e 5d
  6. To find top 20 files with most spacels -ltrawk -F" " '{print $5 $9}' sort -ntail -20
  7. To find record in first file not in second comm -13 testfile1.txt testfile2.txt
  8. If you are looking from something that is contained in a file but you don't know which directory it is in do the following: find . -name "*" xargs grep -i something This will find all of the files in the directory and below and grep for the string something in those files!
  9. Delete Files Delete all the files starting with name testfile find . -type f -name "testfile*" exec rm -f {} \;
  10. Remove blank space from file sed -e "s/ *//g" testfile.txt >testfile.txt_wo_space
4
By
at 4 comments:
Labels: