Building a Programming Language from Scratch - Day 1 (lexer)

Which language humbleOS will ultimately be written in (after I learn enough assembly to make a bootloader) has been a topic of frequent discussion. After much discussion, debate, and thought, I landed on Lonely, the fast, modern, and low-level systems programming language. The only problem? It doesn't exist yet. This series is me creating Lonely.

Day 1:
Built a working lexer that supports all the syntax required for variable declarations, function declarations, function calls, and comments.

[SonikPixelz]

This is both insane and awesome! Keep going my friend!

[jc-aguilar]

This is awesome. For comments, I highly recommend to support markdown from the start.

[asterbloom]

This is awesome! I've got a similar project I'm doing for RISC-V. I actually drafted the entire language already but decided to just continue with at least the lowest level parts of the kernel in C. I discovered the philosophy behind Zig after listening to a few talks and realized these guys were saying the exact same thing I was (except with more money and publicity) and that it actually does everything I want. I don't think I will learn Zig beyond using the compiler. However, I may decide to get back to the language and actually restructure it a bit. I was making a lexer where I last left off. I will say this has been a hell of a journey for myself as a programmer, even just going as far as considering such projects as these.

I'll definitely be following this project of yours, I hope I can learn a thing or two from you 😁

My RISC-V boards just shipped in today so I'm excited to get to work on those. Woohoo!

[OneAndOnlyMe]

Cool start to the project! Lexers are a great way to improve one's programming skills.

Another way to write the token lexer/parser would be as a state machine loop which can simplify having to peek ahead so much. Advance through source string one char at a time, and switch states in the loop. If you hit a alpha-char enter identifier parsing state, if you hit a number then enter number parsing state, and so on for operators, etc. For the tokens I would store extra info like line number with the column position, etc which can then be used to generate useful syntax error messages.

[NotMarkKnopfler]

Take a look at the Forth programming language. No lexer, no decent trees, no BNF. The compiler is the language, and the language is the compiler simultaneously. It even provides a REPL written in itself. Write a line of code, and it is compiled instantly. The cleverest, and simplest implementation you'll ever see. Proper genius.

[rybavlouzi]

Great video mate! I've been wanting to build my own language as well, and you're really inspiring me to do so

[ankush.kapoor]

underrated channel haha keep up the good work!

[oglothenerd]

Building parsers is something I am addicted to.

[xavierthomas]

I would argue that ";" serves a purpose: When parsing an expression it informs the parser that the expression is finished. Without ";" the new line would serve this purpose but then it would affect the way to format the code. Regarding this I greatly prefer non significant white-space because it makes it easier to format the code and comment part of it (temporarily or not).

[RichardSmithers]

What are those popup letters/symbols at the bottom of the screen that appear as he types? Is that a plugin and if so, what is it supposed to do? It looks like random characters.

[RosSokcheanith]

Please do a video about nvim setup and key maps for beginners

[nanpajapi]

custom os? custom compiler? this is gonna be great

[amayesingnathan]

Nice this is similar to a hobby project I've been working on, except I also wanted to create my own custom architecture/hardware design that I would then write an emulator for, and then go on to write an OS and a language for it that compiles to my custom architecture

That first part is taking rather a long time though hahah, so in the meantime I'm definitely going to follow this closely!

Edit: side note, I'm just at the start but I dont love val for const types because val sort of implies something to do with value vs reference semantics

[ahmadgazali]

i wish i started watching this 3 weeks ago

[cthutu]

I prefer walrus (:=) to back arrow (<-) for two reasons: 1) I hate ligatures. Some of them are not readable at all. 2) You can put an optional type between ':' and '=', e.g. 'mut x: int = 42'

[Laflamablanca]

What neovim plugin displays the function signature as you’re typing args into the function?

[abrarmasumabir]

How to get to the point where I can start writing my own programming language. What are the prerequisites?

[EliasNPC]

add possibility for ternary operator or conditional operator

[Nobodyokbro]

Interesting i personally like to use SLY.

[andre]

Let vs Const in JS doesn't have to do with immutability, btw