$ echo xxx\\c
xxx$ { echo abc; echo def; echo ghi; echo xxx\\c; } > y
$ cat y
$ while read line; do echo $line; done < y
$ bash -c 'while read line; do echo $line; done < y'
$ ksh -c 'while read line; do echo $line; done < y'
$ zsh -c 'while read line; do echo $line; done < y'
$ for line in $(<y); do echo $line; done # Preferred notation in bash
$ for line in $(cat y); do echo $line; done # UUOC Award pending
It is also not limited to bash — Korn shell (ksh) and zsh behave like that too. I live, I learn; thanks for raising the issue.
As demonstrated in the code above, the cat command reads the whole file. The for line in `cat $DATAFILE` technique collects all the output and replaces arbitrary sequences of white space with a single blank (I conclude that each line in the file contains no blanks).
Tested on Mac OS X 10.7.5.
What does POSIX say?
The POSIX read command specification says:
The read utility shall read a single line from standard input.
By default, unless the -r option is specified, <backslash> shall act as an escape character. An unescaped <backslash> shall preserve the literal value of the following character, with the exception of a <newline>. If a <newline> follows the <backslash>, the read utility shall interpret this as line continuation. The <backslash> and <newline> shall be removed before splitting the input into fields. All other unescaped <backslash> characters shall be removed after splitting the input into fields.
If standard input is a terminal device and the invoking shell is interactive, read shall prompt for a continuation line when it reads an input line ending with a <backslash> <newline>, unless the -r option is specified.
The terminating <newline> (if any) shall be removed from the input and the results shall be split into fields as in the shell for the results of parameter expansion (see Field Splitting); [...]
Note that '(if any)' (emphasis added in quote)! It seems to me that if there is no newline, it should still read the result. On the other hand, it also says:
STDIN
The standard input shall be a text file.
and then you get back to the debate about whether a file that does not end with a newline is a text file or not.
However, the rationale on the same page documents:
Although the standard input is required to be a text file, and therefore will always end with a <newline> (unless it is an empty file), the processing of continuation lines when the -r option is not used can result in the input not ending with a <newline>. This occurs if the last line of the input file ends with a <backslash> <newline>. It is for this reason that "if any" is used in "The terminating <newline> (if any) shall be removed from the input" in the description. It is not a relaxation of the requirement for standard input to be a text file.
That rationale must mean that the text file is supposed to end with a newline.
The POSIX definition of a text file is:
3.395 Text File
A file that contains characters organized into zero or more lines. The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the <newline> character. Although POSIX.1-2008 does not distinguish between text files and binary files (see the ISO C standard), many utilities only produce predictable or meaningful output when operating on text files. The standard utilities that have such restrictions always specify "text files" in their STDIN or INPUT FILES sections.
This does not stipulate 'ends with a <newline>' directly, but does defer to the C standard and it does say "A file that contains characters organized into zero or more lines" and when we look at the POSIX definition of a "Line" it says:
3.206 Line
A sequence of zero or more non- <newline> characters plus a
terminating <newline> character.
so per the POSIX definition a file must end in a terminating newline because it's made up of lines and each line must end in a terminating newline.
A solution to the 'no terminal newline' problem
Note Gordon Davisson's answer. A simple test shows that his observation is accurate:
$ while read line; do echo $line; done < y; echo $line
Therefore, his technique of:
while read line || [ -n "$line" ]; do echo $line; done < y
cat y | while read line || [ -n "$line" ]; do echo $line; done
will work for files without a newline at the end (at least on my machine).
I'm still surprised to find that the shells drop the last segment (it can't be called a line because it doesn't end with a newline) of the input, but there might be sufficient justification in POSIX to do so. And clearly it is best to ensure that your text files really are text files ending with a newline.
–
–
–
–
According to the POSIX spec for the read command, it should return a nonzero status if "End-of-file was detected or an error occurred." Since EOF is detected as it reads the last "line", it sets $line and then returns an error status, and the error status prevents the loop from executing on that last "line". The solution is easy: make the loop execute if the read command succeeds OR if anything was read into $line.
while read line || [ -n "$line" ]; do
–
–
–
–
Adding some additional info:
There's no need to use cat with while loop. while ...;do something;done<file is enough.
Don't read lines with for.
When using while loop to read lines:
Set the IFS properly (you may lose indentation otherwise).
You should almost always use the -r option with read.
with meeting the above requirements a proper while loop will look like this:
while IFS= read -r line; do
done <file
And to make it work with files without a newline at end (reposting my solution from here):
while IFS= read -r line || [ -n "$line" ]; do
echo "$line"
done <file
Or using grep with while loop:
while IFS= read -r line; do
echo "$line"
done < <(grep "" file)
As a workaround, before reading from the text file a newline can be appended to the file.
echo -e "\n" >> $file_path
This will ensure that all the lines that was previously in the file will be read.We need to pass -e argument to echo to enable interpretation of escape sequences.
https://superuser.com/questions/313938/shell-script-echo-new-line-to-file
# create dummy file. last line doesn't end with newline
printf "%i\n%i\nNo-newline-here" >testing
Test with your first form (piping to while-loop)
cat testing | while read line; do echo $line; done
This misses the last line, which makes sense since read only gets input that ends with a newline.
Test with your second form (command substitution)
for line in `cat testbed1` ; do echo $line; done
This gets the last line as well
read only gets input if it's terminated by newline, that's why you miss the last line.
On the other hand, in the second form
`cat testing`
expands to the form of
line1\nline2\n...lineM
which is separated by the shell into multiple fields using IFS, so you get
line1 line2 line3 ... lineM
That's why you still get the last line.
p/s: What I don't understand is how you get the first form working...
–
Use sed to match the last line of a file, which it will then append a newline if one does not exist and have it do an inline replacement of the file:
sed -i '' -e '$a\' file
The code is from this stackexchange link
Note: I have added empty single quotes to -i '' because, at least in OS X, -i was using -e as a file extension for the backup file. I would have gladly commented on the original post but lacked 50 points. Perhaps this will gain me a few in this thread, thanks.
I had a similar issue.
I was doing a cat of a file, piping it to a sort and then piping the result to a 'while read var1 var2 var3'.
cat $FILE|sort -k3|while read Count IP Name
The work under the "do" was an if statement that identified changing data in the $Name field and based on change or no change did sums of $Count or printed the summed line to the report.
I also ran into the issue where I couldnt get the last line to print to the report.
I went with the simple expedient of redirecting the cat/sort to a new file, echoing a newline to that new file and THEN ran my "while read Count IP Name" on the new file with successful results.
cat $FILE|sort -k3 > NEWFILE
echo "\n" >> NEWFILE
cat NEWFILE |while read Count IP Name
Sometimes the simple, inelegant is the best way to go.
