Sed

Lets practicing using sed to substite text. By default substitutions will allow us to find and replace the first instance of our supplied text per line with the new text we provide. I have included a sample of The Walrus and the Carpenter by Lewis Carroll in this directory that we can use to practice sed on:

cat ./TheWalrusAndTheCarpenter.txt

...

It was so kind of you to come!
      And you are very nice!'
The Carpenter said nothing but
      Cut us another slice:
I wish you were not quite so deaf —
      I've had to ask you twice!'

It seems a shame,' the Walrus said,
      To play them such a trick,
After we've brought them out so far,
      And made them trot so quick!'
The Carpenter said nothing but
      The butter's spread too thick!'

I weep for you,' the Walrus said:
      I deeply sympathize.'
With sobs and tears he sorted out
      Those of the largest size,
Holding his pocket-handkerchief
      Before his streaming eyes.

O Oysters,' said the Carpenter,
      You've had a pleasant run!
Shall we be trotting home again?'
      But answer came there none —
And this was scarcely odd, because
      They'd eaten every one."

For starters lets just try and replace the first instance of a on each line with A:

sed 's/a/A/' < TheWalrusAndTheCarpenter.txt > A.txt
cat A.txt

...

It wAs so kind of you to come!
      And you Are very nice!'
The CArpenter said nothing but
      Cut us Another slice:
I wish you were not quite so deAf —
      I've hAd to ask you twice!'

It seems A shame,' the Walrus said,
      To plAy them such a trick,
After we've brought them out so fAr,
      And mAde them trot so quick!'
The CArpenter said nothing but
      The butter's spreAd too thick!'

I weep for you,' the WAlrus said:
      I deeply sympAthize.'
With sobs And tears he sorted out
      Those of the lArgest size,
Holding his pocket-hAndkerchief
      Before his streAming eyes.

O Oysters,' sAid the Carpenter,
      You've hAd a pleasant run!
ShAll we be trotting home again?'
      But Answer came there none —
And this wAs scarcely odd, because
      They'd eAten every one."

Success! To explain what just happened lets look at the sed command we typed:

sed 's/a/A/' < TheWalrusAndTheCarpenter.txt > A.txt

We write our sed script inside of single quotes. Inside our single quotes we first describe our action; here we wanted to substitute text so we used s. After describing our action we provide a forward slash: /. For a substitution action we then provide the text we want sed to find, in this case: a. After that we provide another forward slash and the text we want to replace our found text with: A. Finally we close out our substitution with one final forward slash and make sure we close out our sed script with a single quote. After our sed script can give a less than sign: < to provide sed with an input file and a greater than sign: > to provide sed with an output file.

To replace every instance of our specified text we can add a g after our closing forward slash to tell sed this is a global substitution:

sed 's/a/A/g' < TheWalrusAndTheCarpenter.txt > A2.txt
cat A2.txt

...

It wAs so kind of you to come!
      And you Are very nice!'
The CArpenter sAid nothing but
      Cut us Another slice:
I wish you were not quite so deAf —
      I've hAd to Ask you twice!'

It seems A shAme,' the WAlrus sAid,
      To plAy them such A trick,
After we've brought them out so fAr,
      And mAde them trot so quick!'
The CArpenter sAid nothing but
      The butter's spreAd too thick!'

I weep for you,' the WAlrus sAid:
      I deeply sympAthize.'
With sobs And teArs he sorted out
      Those of the lArgest size,
Holding his pocket-hAndkerchief
      Before his streAming eyes.

O Oysters,' sAid the CArpenter,
      You've hAd A pleAsAnt run!
ShAll we be trotting home AgAin?'
      But Answer cAme there none —
And this wAs scArcely odd, becAuse
      They'd eAten every one."

Most of the time when we use sed though we won't be providing input and output files. Most of the time when we use sed we will be piping in the output of another command and using sed to alter it:

echo "Hello World!" | sed 's/llo/y/'

Hey World!

You will notice we also only used sed to replace llo with y. This is because sed doesn't have to look for single letters or whole words but it actually looks for text matching a regex we provide. For more information on regex notation look at my guide on regex.

Instead of having sed take an input file and generate an output file we can use the -i flag to inline edit the file provided:

cat Cat.txt
sed -i 's/Meow/Woof/g' Cat.txt
cat Cat.txt

Meow
Woof

Another aspect of sed scripting that allows us to search for specific lines to edit is line patterns. To use line patterns we simply specify the pattern we want to search for before listing our action:

cat TheWalrusAndTheCarpenter.txt | sed '/Walrus/s/said/grunted/'

...

It was so kind of you to come!
      And you are very nice!'
The Carpenter said nothing but
      Cut us another slice:
I wish you were not quite so deaf —
      I've had to ask you twice!'

It seems a shame,' the Walrus grunted,
      To play them such a trick,
After we've brought them out so far,
      And made them trot so quick!'
The Carpenter said nothing but
      The butter's spread too thick!'

I weep for you,' the Walrus grunted:
      I deeply sympathize.'
With sobs and tears he sorted out
      Those of the largest size,
Holding his pocket-handkerchief
      Before his streaming eyes.

O Oysters,' said the Carpenter,
      You've had a pleasant run!
Shall we be trotting home again?'
      But answer came there none —
And this was scarcely odd, because
      They'd eaten every one."

Now every line that contained the text Walrus had the first instance of the word said changed to grunted. Remember though that since we didn't at a g to the end this command only changes the first instance of the word said. If we wanted to change all instances of the word said on any line containing Walrus would need to make it global with: /Walrus/s/said/grunted/g. In this case it does matter since our poem doesn't abuse the word said.

We can also use the line pattern functionality of sed to delete lines that match our line pattern. Let's take a look at that:

sed '/Carpenter/d' < TheWalrusAndTheCarpenter.txt > CutWalrus.txt
cat CutWalrus.txt

...

It was so kind of you to come!
      And you are very nice!'
      Cut us another slice:
I wish you were not quite so deaf —
      I've had to ask you twice!'

It seems a shame,' the Walrus said,
      To play them such a trick,
After we've brought them out so far,
      And made them trot so quick!'
      The butter's spread too thick!'

I weep for you,' the Walrus said:
      I deeply sympathize.'
With sobs and tears he sorted out
      Those of the largest size,
Holding his pocket-handkerchief
      Before his streaming eyes.

      You've had a pleasant run!
Shall we be trotting home again?'
      But answer came there none —
And this was scarcely odd, because
      They'd eaten every one."

Now the Carpenter character doesn't exist in our poem. We just need to use the action d instead of s which stands for delete.

What if we wanted to do multiple sed substitutions in one command. sed allows us to have multiple expressions provided in the same command by using the flag -e before each one. Lets see what that looks like:

cat /etc/shells | sed -e 's/usr/u/g' -e 's/bin/b/g'

# Pathnames of valid login shells.
# See shells(5) for details.

/b/sh
/b/bash
/u/b/git-shell
/u/b/fish
/b/fish

We were able to change all instances of usr to u and all instances of bin to b.

One thing that I need to clear the air on is that we aren't actually forced to use the forward slash: / to separate our inputs in a sed command. You can technically use any character you want that isn't part of your phrase. Normally if people don't use / they will use | or # but it doesn't have to be these, these are just the most common. Lets see how this works:

cat /etc/shells | sed -e 's|usr|u|g' -e 's#bin#b#g'

# Pathnames of valid login shells.
# See shells(5) for details.

/b/sh
/b/bash
/u/b/git-shell
/u/b/fish
/b/fish

You can see we got the exact same output as the section above.

We can use sed to print the lines that contain our provided text. This feature is sort of similar to grep:

cat /etc/shells | sed -n '/usr/p'

/usr/bin/git-shell
/usr/bin/fish

By using the -n flag we can suppress automatic printing of and provide a regex pattern to search for. We can then provide our regex pattern and use the print action: p to only print those lines.

One of the lesser used sed functions is the translate function, executed by using y in our sed script. This will find the characters we provided and replace them with what we specify. This function is very similar to the tr command and you can find my guide on tr here. Lets just see a quick example of this in action:

echo "Hello there world! What a beautiful day!" | sed 'y/aieou/AEIOU/'

HIllO thIrI wOrld! WhAt A bIAUtEfUl dAy!

This replaced every lowercase vowel with its uppercase counterpart.

From here you should have about all the information you are going to need on sed.

Here are some neat tricks that can be done with sed if you are thirsting for more sed information.