Performance profiling OpenStack services with repoze.profile

As OpenStack services mature and see ever larger workloads in production, we have an increasing need to optimize them for performance. repoze.profile (official documentation) has always been my go-to tool for profiling WSGI applications. Profiling a WSGI application is only really different than profiling a regular function call in… »

OpenStack Mitaka Design Summit outcomes for keystone

This is a summary of the discussions, design decisions, goals, and direction that came out of the OpenStack Mitaka Design Summit in Tokyo, Japan (fall 2015) with regard to keystone. Token formats Priti Desai and Brad Pokorny kicked off the technical discussions with an absolutely fantastic deep dive on keystone's… »

Deploying domain-specific identity drivers in OpenStack keystone

As of the Juno release, Keystone supports the ability to back each identity domain with a distinct driver configuration. So, for example, you can back one domain with LDAP, one with your own proprietary driver, and all the rest with keystone's default SQL backend. This has previously been nicknamed multi-domain… »

PEP257: Good Python docstrings by example

Following in the spirit of PEP 8, which is a style guide for Python code itself, the lesser-known PEP 257 establishes similar high-level conventions for docstrings. The module below attempts to illustrate by example. Hopefully this will be easier to grok than reading the PEP itself. Wondering if you're own… »

Peeking inside OpenStack keystone Fernet token payloads

I've been asked several times now how to go about peeking into the payloads of Fernet tokens, rather than just deconstructing the high-level token structure. They're not quite as straightforward as PKI, where a simple Base64 decoding will get you quite far, nor as opaque and lifeless as UUID tokens.… »

Solarized Black

Solarized Black is a fork of Ethan Schoonover's popular Solarized Dark color scheme that makes one minor tweak to the color palette: the dark teal background color (Base03: #002b36) is swapped for black (Base03: #000000). The result is a higher contrast color scheme that's easier on the eyes, especially when… »

OpenStack Keystone Fernet tokens

Fernet is a secure messaging format explicitly designed for use in API tokens by Heroku. They address many of the same problems that OpenStack faces, and make some of the same design considerations that have already appeared in the OpenStack community. They're non-persistent, lightweight, and reduce the operational overhead required… »

The OpenStack Keystone service catalog

The OpenStack Keystone service catalog allows API clients to dynamically discover and navigate to cloud services. The service catalog may differ from deployment-to-deployment, user-to-user, and tenant-to-tenant. The service catalog is the first hurdle that API consumers will need to grok after successfully authenticating with Keystone, making it a critical focal… »

The anatomy of OpenStack Keystone token formats

Tokens in Keystone are generally composed of a number of technologies layered together. All tokens can be deconstructed into at least two layers: a payload which is wrapped in some transport format. The payload provides attributes such as uniqueness, identity, and authorization context. The transport format provides the necessary packaging… »

Benchmarking OpenStack Keystone token formats

tl;dr: PKI and PKIZ tokens are slower than UUID tokens, and based on the June 2015 update, Fernet tokens are faster to create than UUID tokens (but also way slower to validate). The simplest token format in Keystone today is that of UUID tokens: they're randomly generated 32 character… »

Bacon-wrapped jalapeƱo dove poppers

Yep, you read that right. I scoffed at the idea of these a bit at first, because I didn't think any of the dove's flavor would come through after wrapping them in bacon and slathering them in cream cheese, but, to my surprise, the whole thing works really well. I… »

Single- vs multi-tenant clouds

It's hard to standardize on the broader definition of single-tenant versus multi-tenant clouds because everyone seems to disagree on the definition "tenant" (naturally, I'll blame the marketing folks). Everyone can agree that "tenancy" refers to resource isolation. My application does not know that your application exists. My application cannot steal… »

OpenStack Kilo Design Summit outcomes

This is a summary of the discussions, design decisions, goals, and direction that came out of the OpenStack Juno Design Summit in Paris (fall 2014). Unlike my previous design summit adventures, which were primarily focused on Keystone (I'll leave that to Morgan Fainberg to cover), I'm making an attempt to… »

Hierarchical multitenancy

Welcome to the biggest, scariest word in OpenStack. Please don't run away (yet, anyway). Background Keystone's original model for multitenancy was entirely flat: tenants had no relation to one another whatsoever. In Grizzly, we renamed tenants in our API to projects and introduced the concept of domains to serve as… »

Responsibilities of an OpenStack program technical lead (PTL)

This is my perspective on the responsibilities of an OpenStack PTL. These responsibilities are in addition to those which may be delegated to project czars, but it's up to the PTL to ensure that they are all met. Serve as a point of contact to the community. You will receive… »

Responsibilities of OpenStack project czars

Note: This was a model under active discussion during the Juno development cycle (summer 2014), but did not come to fruition in the community. I'm leaving it here for historical reference. While OpenStack PTLs are ultimately accountable for the project as a whole (including the responsibilities outlined below), PTLs have… »

Reviewing code

As a Program Technical Lead for OpenStack, part of my duty involves identifying and supporting "core reviewers" from our open source community: developers believed to be capable of upholding high technical standards and thus empowered to drive the project's direction via gerrit. While each reviewer always brings a differing perspective… »

OpenStack Keystone hackathon outcomes for Juno

Keystone's OpenStack Juno Hackathon in San Antonio, TX (summer 2014) was certainly a productive one. In total, we had 20 Keystone community members in attendance, including 8 core reviewers. We collaboratively revised and merged 5 feature proposals to openstack/keystone-specs, along with 25 additional patches across the identity program's 5… »

Debug vs. Verbose

Debug mode is intended for developers, not operators. Developers need call stacks, line numbers, and raw data to perform stepwise problem isolation. They work on environments that are disposable and do not contain sensitive data. Everything must be precisely and easily repeatable. Operators need insight into the behavior of systems… »

Words of wisdom

Boring systems build badass businesses: Innovate on your core product, not on your plumbing. — Matt Jaynes Programming sucks: That's your job if you work with the internet: hoping the last thing you wrote is good enough to survive for a few hours so you can eat dinner and catch… »