Brendan Gregg

Cloud computing facilitates rapid deployment and scaling, often pushing high load at applications under continual development. DTrace allows immediate analysis of issues on live production systems even in these demanding environments – no need to restart or run a special debug kernel. For the illumos kernel, DTrace has been enhanced to support cloud computing, providing more observation capabilities to zones as used by Joyent SmartMachine customers. DTrace is also frequently used by the cloud operators to analyze systems and verify performance isolation of tenants. This talk covers DTrace in the illumos-based cloud, showing examples of real-world performance wins.

• slides

Note: This is NOT the entire text of the slides – text surrounding/referring to diagrams and other images can be seen in the video.)

whoami

G’Day, I’m Brendan

These days I do performance analysis of the cloud

I use the right tool for the job; sometimes traditional, often DTrace.

DTrace is a magician that conjures up rainbows, ponies and unicorns — and does it all entirely safely and in production!

Or, the version with fewer ponies: DTrace is a performance analysis and troubleshooting tool

Instruments all software, kernel and user-land. Production safe. Designed for minimum overhead.

Default in SmartOS, Oracle Solaris, Mac OS X and FreeBSD. Two Linux ports are in development.

There’s a couple of awesome books about it.

illumos

Joyent’s SmartOS uses (and contributes to) the illumos kernel.

illumos is the most DTrace-featured kernel

illumos community includes Bryan Cantrill & Adam Leventhal, DTrace co-inventors.

Agenda

Theory: Cloud types and DTrace visibility

Reality

• DTrace and Zones
• DTrace Wins

Tools

• DTrace Cloud Tools
• Cloud Analytics

 

Cloud Types

We deploy two types of virtualization on SmartOS/illumos:

• Hardware Virtualization: KVM
• OS-Virtualization: Zones
• Both virtualization types can co-exist

KVM

• Used for Linux and Windows guests
• Legacy apps

Zones

• Used for SmartOS guests (zones) called SmartMachines
• Preferred over Linux:
  • Bare-metal performance Less memory overheads Better visibility (debugging)
  • Global Zone == host, Non-Global Zone == guest
  • Also used to encapsulate KVM guests (double-hull security)

DTrace can be used for:

• Performance analysis: user- and kernel-level
• Troubleshooting

Specifically, for the cloud:

• Performance effects of multi-tenancy
• Effectiveness and troubleshooting of performance isolation

Four contexts:

• KVM host, KVM guest, Zones host, Zones guest
• FAQ: What can DTrace see in each context?

Hardware Virtualization: DTrace Visibility

Host can see:

• Entire host: kernel, apps
• Guest disk I/O (block-interface-level)
• Guest network I/O (packets)
• Guest CPU MMU context register

Host can’t see:

• Guest kernel
• Guest apps
• Guest disk/network context (kernel stack)
• … unless the guest has DTrace, and access (SSH) is allowed

Hardware Virtualization: DTrace Visibility

Guest can see: Guest kernel, apps, provided DTrace is available

Guest can’t see:

• Other guests
• Host kernel, apps

OS Virtualization: DTrace Visibility

Host can see:

• Entire host: kernel, apps
• Entire guests: apps

Operators can trivially see the entire cloud

• Direct visibility from host of all tenant processes

Zooming in, 1 host, 10 guests:

All can be examined with 1 DTrace invocation; don’t need multiple SSH or API logins per-guest. Reduces observability framework overhead by a factor of 10 (guests/host)

OS Virtualization: DTrace Visibility

Guest can see:

• Guest apps
• Some host kernel (in guest context), as configured by DTrace zone privileges

Guest can’t see:

• Other guests
• Host kernel (in non-guest context), apps