This page is a quick tutorial about sed.
sed is a stream editor, i.e a non-interactive text editor that process a stream of text. 90% of the time, sed is used as a very powerful search and replace tool.
Pre-requisites:
Understand shell redirections and pipes (>, <, |).
Understand regular expression
Contents
The "s" (search and rplace) command
A simple search and replace
Let's assume we have a test file named test1.txt containing
Our prefered fruits: I like apples. John likes pears. Elisabeth also likes pears.
Let's instruct sed to edit the file test1.txt, and run the command s (i.e search and replace) to search for "apples" and replace all of them with "oranges", then prints the result:
sed -e "s/apples/oranges/g" test1.txt
Our prefered fruits: I like oranges. John likes pears. Elisabeth also likes pears.
The delimiting character don't have to be the slash (/).
- The "g" after the third delimiter instructs sed to process all occurrences of the search string on the line (that's a common mistake).
Expression delimiter
So we run:
sed -e "s,e,XXX," test1.txt
Our prXXXfered fruits: I likXXX apples. John likXXXs pears. ElisabXXXth also likes pears.
Only the first, lower case, "e" of each line were replaced.
Case sensitivity
if we pass the option i, the matching will become case-insensitive :
sed -e "s,e,XXX,gi" test1.txt
Will replace each and every instance of e or E in the file :
Our prXXXfXXXrXXXd fruits: I likXXX applXXXs. John likXXXs pXXXars. XXXlisabXXXth also likXXXs pXXXars.
Regular Expression
The great thing about sed is that it works with regular expressions :
sed -e "s,[eE],XXX,g" test1.txt
Will replace each and every instance of e or E in the file :
Our prXXXfXXXrXXXd fruits: I likXXX applXXXs. John likXXXs pXXXars. XXXlisabXXXth also likXXXs pXXXars.
It is common so strip leading white space with:
sed -e "s/^\s*//" test1.txt
Will search all the spaces (including tabs, etc.) at the begining of the lines, and replace them with... nothing!
Our prefered fruits: I like apples. John likes pears. Elisabeth also likes pears.
or something more complex...
sed -e "s/^.*like\(s\)\?/I love/" test1.txt
Our prefered fruits: I love apples. I love pears. I love pears.
Now let's assume we have a file test2.txt:
jdoe:Jown Doe:male hmustermann:Hans Mustermann:male ytaro:山田太郎:female pmartin:Paul Martin:male
We want to rewrite strings into something like jdoe is "Jown Doe". To achieve that, we instruct sed to replace the first ":" with the string is ", then we use sed a second time to drop what's after the other ":" (by replacing it with the a double quote ").
sed -e 's/:/ is "/' | sed -e 's/:.*/"/' test2.txt
Which could (and should) be written as sed -e 's/:/ is "/' -e 's/:.*/"/' test2.txt , that's completely equivalent.
jdoe is "Jown Doe" hmustermann is "Hans Mustermann" ytaro is "山田太郎" pmartin is "Paul Martin"
Buffers
The above example isn't very readable/maintainable ... but sed make things easier: It is possible to store different part of a matching search regex into some buffers, then use them in the replace part:
sed -e 's/^\(.*\):\(.*\):.*$/\1 is "\2"/' test2.txt
jdoe is "Jown Doe" hmustermann is "Hans Mustermann" ytaro is "山田太郎" pmartin is "Paul Martin"
Reminder: In regular expressions, .* is globbing (it eats as many characters as it can, so in this example, it is much safer to write : sed -e 's/^\([^:]*\):\([^:]*\):.*$/\1 is "\2"/' test2.txt so the script doesn't get broken if a 4th field is added to the input file.
another example with the same sample file:
sed -e 's/^\([^:]*\):\([^:]*\):\(.*\)$/<a class="\3" href="\/~\1">\2><\/a><br>/' test2.txt
<a class="male" href="/~jdoe">Jown Doe></a><br> <a class="male" href="/~hmustermann">Hans Mustermann></a><br> <a class="male" href="/~ytaro">山田太郎></a><br> <a class="male" href="/~pmartin">Paul Martin></a><br>
Range
It is possible to specify which portion (range) of the file we want to process.
For example, we want to keep the last word of the sentence, but only starting at the second line:
sed -e "2,\$s/^.* //" test1.txt
Our prefered fruits: apples. pears. pears.
The range can be expressed as line number, or based on matching regular expression:
sed -e '/John/,/Elisabeth/s/like/love/' test1.txt
Our prefered fruits: I like apples. John loves pears. Elisabeth also loves pears.
Yes, the syntax is /from/,/to/s/search/replace/arg !
Grep'ing
Often, you only want to print the lines where you have (searched and) replaced text:
sed -n -e 's/like/love/p' test1.txt
I love apples. John loves pears. Elisabeth also loves pears.
we have added the option -n to instruct sed it shouldn't print lines by default, then we use "p" at the end of of the search expression ("s///p"), to instruct sed to print the lines that match that expression.
The "d" (delete) command
sed also has a d (delete} command, which purpose is quite similar to grep :
sed -e '/fruits/d' test1.txt
I like apples. John likes pears. Elisabeth also likes pears.
However, It's useful when you need to choose the range of lines to grep:
Drop the lines between the ones containing John and Elisabeth: sed -e '/John/,/Elisabeth/d' test1.txt
Our prefered fruits: I like apples.
Drop the first line: sed -e '1d' test1.txt
I like apples. John likes pears. Elisabeth also likes pears.
Drop the lines, from the line containing John, until the end of the file sed -e '/John/,$d' test1.txt
Our prefered fruits: I like apples.
Reminder: $ is a special char under unix shell. You need to escape it inside double quoted strings (like "\$"), but not inside single quoted string (like '$').
grep and sed
In most case, you can/should avoid using grep and sed on the same line, since sed can grep files:
Don't write:
cat myfile | grep -E "^[[:blank:]]*MYVAR=" | sed -e 's/^[^=]*=//'
But write:
sed -n -e 's/^\s*MYVAR=//p' myfile
(But there are some exception, especially when grep can discard most of the lines of a large file).
See also
This page is a quick overview of the main sed features. refer to sed official manual (also available as infopage):