Writing My Own Malloc in C

Показать описание

References:

Support:

Feel free to use this video to make highlights and upload them to YouTube (also please put the link to this channel in the description)

Рекомендации по теме

Комментарии

Блин чувак, это божественный канал - спасибо за творчество, хороший звук и подачу. Давненько стримы по программированию искал, так тут еще и понятный, родной акцент :)

egornovikov

Very interesting; I learned a lot about how memory allocation works today

Jaz_

bsearch was work correct. The bug was in the return result. You shouldn't have divided the pointer difference by the chunk size.

freestyle

46:08 "production-ready" allocators store the metadata within the chunks themselves–the size of the chunk, the state of the chunk (in case needed by garbage collection), a pointer to the next chunk, ...etc.
I think this kind of list is known as an "intrusive list" or an "embedded list".

if you're wondering, you can take a look at GCC's implementation of dynamic allocation of memory.
it uses binning where you have bins and each bin has chunks of a certain size (restricted by the bin) and chunks store the metadata using "boundaries", and there are two boundaries, one at the end of the chunk and one at the beginning.
it is pretty interesting!

mstfabideer

Quick note on fixed sized chunk allocators (or pool allocators), you don't need an additional array to hold the reusable old chunks. You can just hold a pointer to the last freed chunk,
and use that "free" memory space to keep a pointer in that chunk to the next one (and so on). They then form a single-linked list containing all re-usable chunks, and you can easily add or remove from the head of the list. Then you just return the head of the list and shorten it by one, or allocate a brand new chunk if the list is empty.

All operations should remain O(1), and the only memory cost is 1 pointer for the head of the list.

tomcutts

You're the best youtuber I've found providing interesting content that's actually useful and helps people learn new things.

SuperMusikstar

Nice one! I don't know if this was already mentioned, but the chunk information is usually saved in the beginning of the allocated chunk. In other words, you always allocate the required size + the size of chunk info structure. That's why you always get unique pointer back even if you allocate 0 length chunk. This approach has multiple benefits in term of performance (less redirections) and reduced memory footstep for the heap management.

necro_ware

I wrote my own malloc implementation some time ago.

My approach was to make linked lists with a next and "up" pointer, a member specifying whether the memory is free, and the amount of memory "managed" by the node. Memory is obtained by sbrk, mmap or malloc (redundant lol) depending on compile flags, but the default is sbrk(n), which extends the program's data segment by n bytes. A left/right list represents contiguous memory, while up/down are separate calls to sbrk and are therefore not guaranteed to be contiguous. At first, a small block (4KiB IIRC) is allocated for the first row, so the size of the memory being managed is 4096 - sizeof (struct memnode). When a memory request is made, we go up the list from the bottom, left to right, bottom to top, to find the first contiguous chunk of memory that can satisfy the request when the bookkeeping node is factored in. So if a request for 10 bytes is made, then there must be 10 + sizeof (struct memnode) bytes available. If so, then the list is bisected. The first node has its size reduced, and after the last byte of its owned memory, a node is made out of the "difference", then the first node is returned and marked as not free. Memory is obtained by adding memnode size, the bookkeeping node is obtained from a pointer in the free function by subtracting memnode size. If enough memory cannot be found before running into the null pointer, then the list head's up pointer is checked. If it's not null, then the same process continues there and so on until either enough free memory is found or a null pointer is reached in the up list. If all lists are exhausted, then a new pool is made with exactly enough memory to satisfy the request and pushed to the top of the uplist.

For freeing, and this is perhaps not the most efficient algorithm, it goes over the whole list starting from the top of the uplist, merges adjacent free nodes, and if head->next == NULL and head->isfree is true, then sbrk is called again to return it to the OS. Since the top of the list is always the most recently allocated, this is safe as long as this malloc is the only malloc that's been used.

IIRC the bottom of the list is a static char array rather than something obtained from the heap as a small optimization, so before free calls the deallocator function, it makes sure the node being freed doesn't have the same address as the head of the bottom of the up list. Memory is re-used bottom-up as it becomes available, although as always, fragmentation is an unavoidable issue. Writing my implementation taught me that syscall overhead isn't the only reason not to be loosey goosey with heap allocation. The other issue is that being all over the place with heap allocations will create fragmentation no matter what implementation of malloc you use, resulting in potentially inefficient memory usage.

So that's how my implementation works. A circular linked list would have probably worked, although to be completely safe, there'd have to be calls to sbrk(0) and comparisons between pools and their addresses/sizes to sanity check that they really are contiguous and that something else hasn't called sbrk in the interim.

As I was writing this, it occurred to me that a next pointer is not even necessary, since the whole list is contiguous. Just add the size of the memory pool to get the location of the next node. I guess that's why they call it a heap, eh? 2n+0 2n+1. Doing this would eliminate the null pointer so the total size of the heap would also have to be tracked. Something like

struct memnode {
size_t size;
uint8_t flags;
};

static char *pool;
static size_t poolsize;

struct memnode *
memnode_next(struct memnode *node)
{
void *ptr = node;
size_t limit = poolsize - (ptr - pool)

ptr += sizeof *node + node->size;

if (ptr - node > limit)
return NULL;

return ptr;
}

It'll be interesting to see how someone else does it.

BradenBest

This is the only video that has a clickbaity title, but then the content is even more extreme than the title

BaptistPiano

These kinds of exercises is why I’m finding that I’m getting better at understanding C and how the computer operates at a fundamental level!!

BigJMC

Awesome!!! The memory manager is an important thing that often requires customization depending on the specifics of the project. You can't trust the compiler everything.

RobotN

About that "freed" or "freeed" question, we somehow have this in french.

The verb "to create" => "créer", in feminine form (we generally add another e) => "créée". It's uncommonly used but it exists and is grammatically correct.

Generally people just write "crée" which confuses everything, but we're used to being confused :)

phyl

i regularly use a memory pool class i made. has dynamic width blocks and chunks, a partition table, automatic defrag, supplies smart pointers, garbage collection, and more. one of the most useful things ive ever made. can use it on every project for fast memory access and no worries about clean up. definitely recommend using one

slkyslm

Very interesting, thanks for sharing!

greob

A few times i was thinking "no don't do it that way, that'll cause a problem later" and then you get to that point and everything you've set up perfectly solves the problem i was thinking about. It's like well told story

robmckennie

Wrote this as a system design course assignment. Pain in the ass.

cctes

This video inspired me to get back to C and unmanaged languages in general.

A different approach to the problem that, in my opinion, results in simpler code would be to have only one chunk list, and add a "state" variable to Chunk. It's type would be an enumeration, ChunkState with 2 possible values : CHK_FREE and CHK_ALLOCATED.
That way, you don't have to perpetually move chunks across lists.

cover

In chunk_list_find, you should replace `(result - list->chunks) / sizeof(list->chunks[0])` just with `result - list->chunks`

gshinevon

All metadata about allocated and free block stuff is kept inside a block in the same heap. You don't need separate memory to track this information. You just alloc block info + size data and put block info data, then return mem + sizeof(block info) pointer back to the app. The free list requires only the head pointer to the first free block, and all free blocks chained in a singly linked list where you store next pointer in a block info.

dnkreative

dude, knowing this stuff after studying it super hard, and seeing him basically derive everything with a few minutes of thought is crazy. Like how he figured out that the free function should merge the consecutive free slots, without even having encountered an error at all

LucasSantos-bgjo

Writing My Own Malloc in C

Writing My Own Malloc in C

Malloc Explained in 60 Seconds

What if I try to malloc WAY too much memory?

what's in a heap? simple malloc implementation internals explained

Making a simple Dynamic Memory Allocator (malloc)

Writing our own malloc (ep. 5)

What does it take to design a memory allocator? Dynamic Memory Allocation System Design

The Origins of Process Memory | Exploring the Use of Various Memory Allocators in Linux C

Code Review: My own malloc() function in C (4 Solutions!!)

Implement an aligned malloc function using the built-in malloc function

coding in c until my program is unsafe

One reason to Modify your Memory Allocator (C/C++)

GWU OS: memory allocation - malloc and free

I made the same game in Assembly, C and C++

How to use malloc to dynamically allocate memory

Simple Malloc Implementation

Memory pool allocator in C without using built-in malloc() and free() functions

Writing Garbage Collector in C

GLibC Malloc for Exploiters: Leak It, Write It, Become a Wizard - Yannay Livneh

My malloc() in C using mmap()

How to write your own code libraries in C.

Dynamic memory allocation || malloc() in c || C language [tutorial]

Code Review: Custom malloc implementation

you can become a GIGACHAD assembly programmer in 10 minutes (try it RIGHT NOW)