Linux Performance Tools, Brendan Gregg, part 1 of 2

Tutorial by Brendan Gregg of Netflix for O'Reilly Velocity conference 2015 Santa Clara. Part 1 of 2.

Description: "

There are many performance tools nowadays for Linux, but how do they all fit together, and when do we use them? This tutorial explains methodologies for using these tools, and provides a tour of four tool types: observability, benchmarking, tuning, and static tuning. Many tools will be discussed, including top, iostat, tcpdump, sar, perf_events, ftrace, SystemTap, sysdig, and others, as well observability frameworks in the Linux kernel: PMCs, tracepoints, kprobes, and uprobes.

This tutorial is updated and extended on an earlier talk that summarizes the Linux performance tool landscape. The value of this tutorial is not just learning that these tools exist and what they do, but hearing when and how they are used by a performance engineer to solve real world problems — important context that is typically not included in the standard documentation."

[domaincontroller]

07:08 system is running slow, commands
07:11 top (10)
07:31 iotop
07:45 iostat -xz 1 (6)
07:57 netstat
08:14 ss
08:34 dstat
08:43 sar -n DEV 1 (8)
08:59 vmstat (3)
09:10 | 09:49 strace
10:17 what do they mean by the system is slow, latency
11:51 packages, sysstat, procps, coreutils

12:07 Anti-methodologies

12:13 street light anti-method
13:25 drunk man anti-method
13:59 blame someone else anti-method
14:43 | 16:02 problem statement method

16:30 workload characterization method
17:48 USE method, utilization, saturation, errors
18:58 USE method for hardware systems
21:20 off-CPU Analysis
22:36 CPU profile method
24:26 RTFM Method
24:54 reading linux source code, jvm
26:50 How do you measure them

27:07 observability tools

27:18 uptime (1)
30:02 top (or htop)
31:57 ps -ef
32:28 vmstat 1 (3)
33:28 instant-xmdz
36:02 mpstat -P ALL 1 (4)
36:14 free -m (7)
36:46 latency is much higher, can you debug it
42:32 observability tools: basic

intermediate

42:48 strace
44:21 tcpdump
46:15 net stat, nicstat
46:49 pidstat 1 (5)
47:21 swapon
47:33 lsof
47:58 sar (8)
50:38 app is taking forever
50:45 | 51:17 pidstat, system time
50:57 system time, iostat
51:26 am i swapping ? vmstat
51:51 strace

[TirajAdikari]

I'm troubleshooting a java performance issue at the moment and found this systematic method of analysis very useful. Thanks Brendan.

[Locane]

Oh hey! You're the author of that observability tools image I have copied to every possible device and storage location so I can always reference it! Hell yeah, thank you!

[bhanureddy]

This is amazing. true love for the bits and bytes

[OryBand]

One of the best devops presentations out there. Good job Brendan! You're making the world a better place one step at a time.

[ruoxizhang]

So clear and educational, thanks for sharing!

[ks_]

Nice vid, reinforced some topics for me and learned new stuff. Next time I’ll be more prepared for my Netflix interview 😅

[jaashutosh]

Very Useful content, thank you Brendan

[abdullahadel]

Thank you for a wonderful session, much appreciated. 👏

[supremepancakes]

I like how you start off straight to the point when you said that when you are mentored by someone you basically eliminate what you dont need to know lol

[davidleal]

Thank you man, I will definitely copy yours methods

[harshbhadoria]

Hi guys, can anyone tell how is it possible to collect all metrics from Linux and visualize them in windows OS? How to make a bridge or connection between Linux and Windows and what are the technologies to be used.

[pixelPlex]

Which tools are used for monitoring network usage/performance of a process?

[yunuskoning]

Hi Brendan,
You are not by any chance available for freelance performance trouble shooting projects?

[QuadDerrick]

"more than 80 characters is a sin in unix world" :):):) didnt know but yea, makes sence.
Norway

[sealovers]

they present linux but all of them use MacOS