Unix Pipeline (Brian Kernighan) - Computerphile

Just what is a pipeline in the computer science sense? We asked Computer Science guru Professor Brian Kernighan

This video was filmed and edited by Sean Riley.

[fdk]

Brian Kernighan should have his own Youtube channel, I'm sure he has many interesting UNIX and general computer history stories to share.

[trylleklovn]

I just love hearing Brian Kernighan talk about this stuff. There is an awesome video on youtube called "The UNIX Operating System" on AT&T Tech Channel from the 70's where a younger Brian Kernighan descripes the exact same principles of UNIX, back when they were still new and groundbreaking - I recommend that video to anyone.

[xeigen]

Pipes are so elegant that they seem obvious and trivial. But that's often the way with great ideas!

[goyabee]

My favorite use of pipeline in my own experience was when I when I piped arecord (to create a pcm data stream from microphone input) into oggenc (to compress the pcm into a vorbis audio stream) into gpg (to encrypt the audio stream(probably should have used ssl but it worked albeit with a delay for entering the passphrase)) into netcat (to send the encrypted audio stream over a network) and on the other machine the reverse ( netcat | gpg | oggdec | aplay ) and effectively created a one-way encrypted voice-over-ip system with just a handful of popular commands.

[compu]

As a system admin I use pipelines all the time. It's so useful to take existing programs and simply string them together. It also makes developing the command / program easier since you can test each step before tacking the next operation on.

[The_Conspiracy_Analyst]

You're redirecting output with a system call pipe(), which is used in the shell you're using. Your shell is the parent that instantiates the child, which is the program your executing (grep, etc). So for all you curious out there, you can do this from within your C programs, it's not just for fun on the shell.

[dice]

It's the simple things like this in Unix that make the system great as a whole.

[THB]

When I saw pipes, I fully understood the magic of unix.

[shadowmil]

Pipes are amazing BTW. Most people who use them take them totally for granted, including me.

[TheHoaxHotel]

It's a Unix system, I know this!

[Guillaurent]

Great vid.
One fundamental piece of information missing in the video is that all those programs you glue together are in fact working at the same time in parallel. So if some of those are CPU hungry, it helps spread the load. And the time needed for the complete chain to complete is basically the time needed for the longest program to complete (except when you are using programs that need the whole data set to do their job, like "sort").

[johnflano]

Very subtly placed copy of "The C Programming Language", maybe a salute to its co-writer.

[ruthklein]

Brian -- Hello. It's Ruth Klein. I worked in Berk Tague's Dept. at Murray Hill. I may have some insight as to how Doug McIlroy's idea of streaming together programs actually got implemented. I believe it came about because of a unique characteristic of the PDP-11 machine language. Before the PDP-11, computer machine languages had specialized instructions to deal with each type of I/O device. For example, there were specialized registers and a specific machine language instruction to send output to a printer. Another set of registers and another machine language instruction would have been used if you wanted to send the same data to a disk file, etc. The PDP-11 got rid of these device dependent machine language instructions. Instead, the PDP-11 replaced these with a single READ and a single WRITE machine language instruction. The actual address of the device would have been put into a machine register used by the READ or WRITE instruction. Bottom line, this meant you could write a device independent program and, when you went to run the program, you could provide parameters as to which input and output devices to use. When DEC announced the PDP-11 computer, they hosted a presentation in NYC. I was the one sent to attend that presentation and found their new machine language method of dealing with I/O devices to be a simple and elegant design. Back at the Labs, I wrote a short memo describing DEC's new mini-computer and this simplified method of dealing with I/O devices. Two days after this memo was distributed, Doug McIlroy stopped by my office, waving the memo, and saying "Tell me more about this". I believe it was this feature of the PDP-11 design that provided the straightforward method of implementing standard input and standard output. And, linking the standard output of one program to the standard input of another program implemented the concept of the pipe.

[DragoniteSpam]

I just realized why the | symbol is called the "pipe" o_0 Unix is awesome.

[goyabee]

Funny how they misinterpreted his mention of using grep, wc, and sort. He meant those are programs that you would frequently pipe data into, not that you would ever use them in a three-stage pipe as was shown in the animation, as that would make no sense at all, wc only outputs a count of lines, words or characters, none of which would make any sense to pipe into sort.

[FEARbraveheart]

This is in my opinion the most compelling reason to script in bash or sh. I love the simplicity of which I can test each element individually.

[hemanthkumarHere]

The first ever programming book that got into my hands was written by Kernighan and Rithcie. Its a pleasure to learn from the pros.

[arthur_p_dent]

I could listen to Brian talk for hours.

[RandomNUser]

This video has the perfect timing. I've got an exam on linux commands on the near future, pipelines included. Understanding the basics and the origins makes it so much easier.

Thanks Computerphile!

[andljoy]

Do one thing and do it well.
Write Programs that work together
Write programs to handle text streams, because that is a universal interface