Lexical Scanning in Go - Rob Pike

Lexical Scanning in Go, a talk by Rob Pike at Google Technology User Group given on Tuesday, 30 August 2011.

[RayPereda]

This is an exquisite example of clean code. Lexical analysis code is often hairy but as Rob shows doesn't have to be. Go first class functions and slices really shine here.

[lukes]

My favourite line: “So we should write our own [lexer], because it's easy, right? Anybody can write code—especially programmers.” (11:45)

[kalekold]

This is now my ideal way of writing a lexer in Go. It's such an elegant solution.

[atanasdenkov]

Beautiful lecture, beautiful language, beautiful programming style.

[seltsamerzeitgenosse]

09:26 "We could use a tool. Lex is a pretty famous one. Initially written by Mike Lesk and then redone when he was an intern by someone called Eric
Schmidt." ... who at the time of this talk was his Boss (and the CEO of Google)

[bokwoon]

So instead of the state machine being implemented as a state variable + a switch statement, it is implemented as functions returning other functions. If state A transitions to state B, it is encoded as function A returning function B.

[joshring]

Really appreciate how this talk made state machines more accessible to normal humans :)

[vonschlesien]

@henriquedante Unicode is actually not a problem - Go as a language natively supports utf8 as its character encoding for the string type. For example, around 23:00-24:00, where you see "switch r = l.next() {...", r is the next rune in the output - i.e. the next unicode code point assembled through utf8 decoding. If you want to accept only upper case, just call the relevant test function from package unicode on the value of r :-)

[rettberg]

Is anybody familiar with the work of Michael Jackson? (sees a lot of hands) Good (goes right into talking about some obscure CS researcher.)

This man is the pinnacle of nerd. I'm pretty sure the audience thought you were about to make a joke about thriller.

[olaifaoluwadarasimiibikunl]

Wow! Awesome talk. Can't wait to try this out. Thanks Rob.

[stovechan]

51:06 They wanted him to stay, be he had to Go. Should he stay or should he go? He decided to Go.

[DamienPollet]

I think we did something like that as an exercise, except each state function would directly call its successors using tail-recursion. That would have been in OCaml, IIRC…

[shakerlakes]

I love the slam against Perl at 4:32. 😉

[naikrovek]

I like how Dave Chaney (now something of a Golang celebrity) is in the audience asking questions. Looks like him, anyway.

[jnevercast]

Master of Goroutines, still can't schedule :p

[ilcorion]

Is there any "Parsing in GO", describing the next step -- using the channel of items in order to build syntax tree?

[VickyGYT]

These talks are so much better than any Cpp++ con talks...

[EugeneCrosser]

@jimmyrcom I see. Honestly, in this particular case it's just a matter of personal preference, the for-loop is "elegant enough" too.

[EugeneCrosser]

I wonder if Go has tail call optimization. You might implement the construct described in minutes 14-15 as a tail call instead of a for-loop.

[henriquedante]

@vonschlesien You're right I didn't think the test functions would be so efficient as they are