Analyzing and manipulating hidden characters in text files

When we encounter strange, unexplainable problems with text files, hidden characters may be reason. This article describes several possibilities to tackle line-ending and whitespace problems.

Correcting mixed line endings

If a file has mixed line endings, the standard tool flip may help you:

A check with file reveals that the file test.txt has mixed line endings. Flip unifies the line endings to Unix (-u) or Windows (-m) standard:

Examining files

vim can show whitespace characters, if you enable the option list. In command mode, execute the following to show whitespaces like tabs or line endings. Unfortunately, the editor does not differentiate between different types of line endings.

Use :set nolist to return to normal view. With :set ff the program identifies the line ending standard.

If you need to get a detailed picture of the whitespace characters in your document, the octal file viewer od may be helpful, it displays the file as octal values and (interpreted) ASCII characters:

The results looks as follows:

Using cat -v text.txt, you can see bogus (non-Unix) line endings  being marked with a special symbol: ^M

Leave a Reply

Your email address will not be published. Required fields are marked *

Please type the characters of this captcha image in the input box

Please type the characters of this captcha image in the input box