Coding Challenge #49: Photo Mosaic with White House Social Media Images

References:

Videos:

Related Coding Challenges:

Timestamps:
0:00 Introduce the coding challenge
0:24 Mention the 'White House Social Media Data Visualization' coding challenge
0:38 Introduce the dataset
1:08 Explain the goal of the coding challenge
1:39 Set up a Processing project
2:00 Draw an image
2:42 Explain how to pixelate the image
3:00 Copy the image at a smaller scale
4:36 Loop through the pixels of the smaller image
5:30 Consider the color of each pixel
6:03 Draw a larger rectangle with the same color
7:02 List files from a local directory
9:08 Filter files to consider only images
11:00 Load all images
11:42 Test code with a subset of images
12:25 Consider the brightness of the pixels instead of color
12:54 Explain the logic behind the mosaic
13:10 Consider the brightness of the images
14:05 Create an array of brightness values
14:20 Compute the average brightness of each image
15:24 Store the average brightness
17:02 Create an array of possible brightness values
17:17 Associate values to the image with the closest average brightness
20:06 Draw images instead of rectangles
21:11 Test code
22:10 Explain `OutOfMemoryError`
22:54 Sample images down
24:52 Suggest improvements and creative possibilities
26:40 Conclude the coding challenge

Editing by Mathieu Blanchette
Animations by Jason Heglund
Music from Epidemic Sound

#pixelarray #imageprocessing #java #processing

[jesseserrano]

I used to be afraid of coding and thought only super-smart people very good at math could do it but I've been watching so many of your videos I'm not only understand coding, I find myself wanting to change my major to learn even more! Thanks man

[starrycavereal]

I found your channel and am binge-watching now like nobody's business

[holdengreene]

If you dont like your code or you think you could be solving the problem better, I know i would rather see you take a little more time and show how you think the problem should be solved. Watching you write code that you constantly undermine makes me feel like whatever I learn is actually just a bad habit. That doesn't mean that you need to overly complicate with bloated solutions but watching you write simple elegant functional code that demonstrates good practices. Love your channel, keep up the great work!

[samuelgoncalves]

You're so awesome. I got in love with Generative Art because of you. Many thanks for sharing your knowledge !

[lucaxtal]

Really Nice!!! After watching this video I couldn't wait to try it by myself in JAVA, and the result was nice. In Grayscale, like you did in the tutorial is not good because of many repetition of images (just little set of colors) but when I added color to the image mosaic it was wonderful. Just a little tricky to understand how to do it. My implementation was this: based on your tutorial I did avgRed, avgGreen, avgBlue for the tile of mosaic, and have 3 values. The candidate pixel to be replaced also has R, G, B so by evaluating the diff between candidate pixel(R, G, B) and tileAvg(R, G, B) with euclidean distance I could choose good Pixel replacement candidate between the Tiles. (the tiles are the images you loaded). Maybe is not clear but I can give some code if someone is interested.

[Drivr]

Hey, Daniel, I really appreciate what you do. Love the enthusiasm you put into your videos!

[ericboyer]

Thanks Daniel, for the positive way you share your approach of coding, what to do from a start and what is open next with p5 and processing using it. I keep on track everyday whatever the subject, cause i find it always amuzing and a start of doing some other stuff using your material. Somehow there's plenty of fancy stuff to do, it's a never ending discovery, i just like that.Thanks Daniel.

[tw]

store a brightness to path mapping and load images lazily when you need them

[vlada]

Man I can't explain how much I luuv to watch your videos, I get really motivated to code :P !!

[hjjol]

thx, for illuminate my morning with your energie, and happyness ! 7:49 this is what i want !

[DMSG]

I wrote a program years ago doing something similar. Your video made me think about the problem again and try out a couple of things. Here's what I found:
First, I want to define some terms that I will use in the descriptions below:
- tile, a rather small, square picture that is used as picture element in the output picture.
- target picture, a downscaled version of the picture to match. It's the "target" for the matching algorithm.

[edit: in the links below, I don't know what language the site uses for you, but if you click on the button in the center of the bigger pictures, the full resolution is loaded.]

1) We need the tiles to be square and all the same size, so that really screams for preprocessing...

2) Dealing with color is really difficult, because to do it properly you have to deal with color spaces etc. But I found that for this purpose, it's OK enough to assume linear color spaces even though most of them are actually non-linear. So let's assume that the difference of e.g. an intensity in red between 226 and 227 is exactly as high as the difference of an intensity in red between 120 and 121. Also assume that a difference in e.g. the red channel affects the color just as much as the same difference in the blue or green channel (which is also not true in most color spaces).
Then every color can be represented as a point in the three-dimensional color space with red, green and blue as its axes. The "error" of a pixel color is then the distance between the point for the pixel color in the target picture and [edit: the point for] the pixel color in the tile that should match. The distance is (under the assumption of linear color space): err = square root (dr*dr + dg*dg + db*db), as described by the Pythagorasean theorem, where dx is the intensity difference in color channel x.

Every tile gets downscaled to 1x1 pixels.


5) Much better already, but there's still whole areas in the output picture that use the same tile over and over again, so let's not use the single best match, but a random tile of the best *n* matches. I found n=10 is pretty good (but that depends on the number of tiles the algorithm has at its disposal). In fact, let's not pick one with uniform distribution, but prefer tiles with a better match. I used a very simple approach that behaves quite nicely: choose two random values within the range and pick the smaller one (smaller indexes in the array refer to smaller errors in my implementation). With this algorithm, the complexity for choosing a random value is still constant, and higher indexes are picked with linearly decreasing probability.

Finally, the results obviously get better the more tile pictures you provide. Soon, inspecting every tile for every pixel (cluster) in the target is impractical. With about 36000 tiles, the matching for something about the dimensions of the examples above took over 15min. Whereas now, the matching with 54000 tiles took only 20sec. That's the matching only, not the assembly and writing to disk of the output picture. So what did I do? I created a database that contains for every tile the best *n* matches for similar tiles. And I use the top 50. The matching starts with testing against 50 tiles that are most different from each other (sampled beforehand). The idea is to get the general direction, is this a red area or a blue one, is ot rather dark or bright? Then the algorithm looks at the "neighboring" tiles of the winner (looked up from the database) to find a better match. And continues to do so as long as a neighbor is better than the current best. With this approach, on average only ~170 tiles are actually tested until the final match was found. That's 170 out of 54000. The creation of the database took quite some time, though, and actually, I sought for the best 1000 matches and created a reduced-to-50 version for testing. But the results have been so convincing that I stuck with it.

What I think would improve results further, is to block used tiles for its neighbor positions only. But that's still to be done...

[ambientsoda]

I think your channel is brilliant, as you introduce idea or are a research source for understanding and making own versions of code😍

[afonsocrg]

You can start for loops with the value of 1 if you want! Awesome video <3

[dennishansen]

Awesome challenge. I didn't realize image processing was so approachable!

[CalzVideos]

If youtube doesn't work out, you could always go into the comedy business. 😀

[maxtaniepetitdol]

You share with us so much tuto, that I've never seen just Cool.

[skaNinja]

Hey Daniel! I have been watching your videos for some time now, being mostly a voyeur who doesn't interact much. However, I have a coding challenge for you (I am not sure where to post this, so here will do)! Curve shortening flow! Have the user draw a closed shape in your window/canvas, and create a program that minimizes the perimeter of said shape while keeping its area the same! Thanks for reading this, and I look forward to your future videos!

[aaditya]

coding train you are awesome man ❤️❤️❤️❤️

[glenneric]

If you had a selection of images at each brightness you could do some image to image correlation to actually select the best image to use for each pixel. This would let you effectively 'borrow' detail and give you a sharper final product.

[Splatpope]

also you simply need to streamline the image processing part, loading each image one by one, scaling it down inside a new image, then remove the original from memory