Description
Rainya is the software development manager for deployment automation in the Product Infrastructure segment at Rackspace Hosting, Inc. Rackspace is based in San Antonio, Texas, United States
Rainya Mosher, Software Dev Manager
Infrastructure Deployment Systems
Twitter: @rainyamosher | Email: [email protected]
Operating OpenStack at
Rackspace for a Hybrid Cloud
Experience
20 March 2014
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Name: Rainya Mosher
• Twitter: @rainyamosher
• IRC Freenode: rainya
• Email:
[email protected]
Rainya is the software development manager for deployment
automation in the Product Infrastructure segment at
Rackspace Hosting, Inc. Rackspace is based in San
Antonio, Texas, United States.
2
Your Presenter
Operating OpenStack @ Rackspace
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
“If computers of the kind I have advocated become the computers of the
future, then computing may someday be organized as a public utility just
as the telephone system is a public utility … The computer utility could
become the basis of a new and important industry.” - John McCarthy,
American Computer Scientist, MIT Centennial, 1961
3
Brief History of Computing
Operating OpenStack @ Rackspace
Mainframe
1960s
Virtualization
1990s
Cloud
2000s
Hybrid
2010s
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
4
Hybrid Cloud Experience
Operating OpenStack @ Rackspace
The hybrid cloud is the notion that an individual can consume whatever
piece of infrastructure they need at the time that they need it with a
consistent experience so they can focus on their core competency, be it
coding or research or data analysis.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Powered by OpenStack
• Open source software with global
community of over 1,000 contributors
and 80 companies
• OpenStack provides freedom from
proprietary software and vendor lock-
in
• Rackspace Infrastructure includes the
following OpenStack services
• OpenStack Compute
• OpenStack Image Service
• OpenStack Networks
• OpenStack Storage 5
Public Cloud at Rackspace
Operating OpenStack @ Rackspace
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
6
OpenStack Cloud Operating System
Operating OpenStack @ Rackspace
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Kickstart
• Install hypervisor
onto physical host
Bootstrap
• Prepare hypervisor
and create control
plane nodes to run
OpenStack
services
Maintain
• Update OpenStack
services in control
plane nodes
7
Building an Open Cloud
Operating OpenStack @ Rackspace
The hypervisor is
where the customer
instance / server is
created.
Control plane nodes are
virtual instances that run
the OpenStack services.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Hypvisors (HVs) &
Compute Control Nodes
Hundreds of HVs &
Cell Control Plane Nodes
Thousands of HVs &
Region Control Plane Nodes
Tens of Thousands of HVs &
Control Plane Nodes
Global
Cloud
Region Region
Cell Cell Cell
HV HV HV HV HV HV
Cell Cell
Region
8
Maintain at Cloud Scale
Operating OpenStack @ Rackspace
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Code
Package
Deploy
Verify
9
Updating the Control Plane
Operating OpenStack @ Rackspace
During a control plane
update, customer
instances on the
hypervisor stay up and
running.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
10
Code Iterations & Releases
Operating OpenStack @ Rackspace
6-Month OpenStack Release Cycle
60 Days
60 Days
60 Days
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Benefits and Challenges of Trunk Deploys
Operating OpenStack @ Rackspace
11
Why We Do It (Benefits)
• Issue Resolution
– Early detection of issues and conflicts
– Shorter feedback loop within the
community
– Faster resolution of issues
• Early Feature Delivery
– Smaller, incremental periodic releases
– More stable release candidates for the
community at end of cycle
Why It’s Hard (Challenges)
• Code Management
– Merge conflicts with local patches
– Disruptive DB migrations
– Service restarts
– Temporary version skew
• Testing
– Devstack-based testing vs testing at
scale
– Rework when issues found in RAX deploy
pipeline
• Process
– CI/CD vs 6-month Releases
– Time to merge upstream patches
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• per-service venv
• .tar of service
venvs + configs
Package
• seed .torrent
• distribute configs
• verify completion
Distribute
• switch version
• sync databases
• apply configs
• verify completion
Execute
12
Package and Deploy
Operating OpenStack @ Rackspace
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Deploy and Verify
Operating OpenStack @ Rackspace
• pre-code
check-in
validation
Dev
• smoke tests
• unit tests
Integration
• functional tests
• integration
tests
QA
• regression
tests
• build tests
Pre-Prod
• smoke tests
• build tests
Production
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Jenkins
Ansible
Puppet
• Execution
• Orchestration
• Configuration
Management
14
Tools to Maintain the Cloud
Operating OpenStack @ Rackspace
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
15
Tools to Maintain the Cloud
Operating OpenStack @ Rackspace
• Puppet
– Configuration management for all control plane nodes
– YAML-defined fact files for config variables
– Masterless configuration to prevent bottlenecks at scale
• Ansible
– Playbooks for orchestration of OpenStack service bootstrap and updates
– Playbooks for orchestration of out-of-iteration hotpatches and changes
• Jenkins
– Scripts called to create virtual environment (venv) packages
– Jobs configured for automating the pull and merge of upstream code
– QE test automation triggered after deployments
– Ansible playbooks invoked from jenkins to reduce human error
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• The hybrid cloud concept will enable individuals to access the
infrastructure they need when they need it.
• One part of the hybrid cloud portfolio at Rackspace is the public cloud,
powered by OpenStack.
• OpenStack is an open source cloud operating system with distinct
services that work together to enable the creation of cloud servers.
• Rackspace uses a variety of tools and processes to maintain
OpenStack services throughout the OpenStack release cycle.
16
Final Thoughts
Operating OpenStack @ Rackspace
Twitter: @rainyamosher | Email: [email protected]
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Backup Slides
Operating OpenStack at Rackspace
17
Many of these backup slides were first presented on 4/16/2013 during the
OpenStack Summit session “Deploying from OpenStack Trunk” and are
included here for reference.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
18
OpenStack Release Cycle
Operating OpenStack @ Rackspace
6-Month OpenStack Release Cycle
new features accepted feature freeze
Planning
Community designs,
discusses, & targets
release cycle
Implementation
Community makes
changes to OpenStack
code & creates new
functionality
Pre-Release
Community focuses
on bug fixes, docs,
& testing
Maintenance
Security vulnerability fixes are supported for the last three Releases
Final
Release
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
19
Rackspace Iteration Cycle
Operating OpenStack @ Rackspace
Choose Release
Branch
Pull upstream code,
merge with internal
patches, and deploy
to CI environment
for validation
Release Branch
Integration
Increase test coverage on
release branch candidate
to include full end-to-end
integration testing with
Rackspace services
Approve Release
Once testing
passes, release is
approved and
production deploy is
scheduled
60-Day Rackspace Iteration Cycle
code may change release branch frozen
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
20
Merge and Branch Strategy
Operating OpenStack @ Rackspace
• Rackspace Development branch is
about 40 patches on top of
OpenStack trunk for internal service
compatability
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
21
Scale of Deploy Pipeline
Learning to Scale OpenStack
1,000s of Nodes 100s of Nodes 10s of Nodes DevStack
Dev
Integration
& QA
PreProd Production
22
RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218
US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM
RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM
doc_236095834.pptx
Rainya is the software development manager for deployment automation in the Product Infrastructure segment at Rackspace Hosting, Inc. Rackspace is based in San Antonio, Texas, United States
Rainya Mosher, Software Dev Manager
Infrastructure Deployment Systems
Twitter: @rainyamosher | Email: [email protected]
Operating OpenStack at
Rackspace for a Hybrid Cloud
Experience
20 March 2014
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Name: Rainya Mosher
• Twitter: @rainyamosher
• IRC Freenode: rainya
• Email:
[email protected]
Rainya is the software development manager for deployment
automation in the Product Infrastructure segment at
Rackspace Hosting, Inc. Rackspace is based in San
Antonio, Texas, United States.
2
Your Presenter
Operating OpenStack @ Rackspace
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
“If computers of the kind I have advocated become the computers of the
future, then computing may someday be organized as a public utility just
as the telephone system is a public utility … The computer utility could
become the basis of a new and important industry.” - John McCarthy,
American Computer Scientist, MIT Centennial, 1961
3
Brief History of Computing
Operating OpenStack @ Rackspace
Mainframe
1960s
Virtualization
1990s
Cloud
2000s
Hybrid
2010s
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
4
Hybrid Cloud Experience
Operating OpenStack @ Rackspace
The hybrid cloud is the notion that an individual can consume whatever
piece of infrastructure they need at the time that they need it with a
consistent experience so they can focus on their core competency, be it
coding or research or data analysis.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Powered by OpenStack
• Open source software with global
community of over 1,000 contributors
and 80 companies
• OpenStack provides freedom from
proprietary software and vendor lock-
in
• Rackspace Infrastructure includes the
following OpenStack services
• OpenStack Compute
• OpenStack Image Service
• OpenStack Networks
• OpenStack Storage 5
Public Cloud at Rackspace
Operating OpenStack @ Rackspace
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
6
OpenStack Cloud Operating System
Operating OpenStack @ Rackspace
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Kickstart
• Install hypervisor
onto physical host
Bootstrap
• Prepare hypervisor
and create control
plane nodes to run
OpenStack
services
Maintain
• Update OpenStack
services in control
plane nodes
7
Building an Open Cloud
Operating OpenStack @ Rackspace
The hypervisor is
where the customer
instance / server is
created.
Control plane nodes are
virtual instances that run
the OpenStack services.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Hypvisors (HVs) &
Compute Control Nodes
Hundreds of HVs &
Cell Control Plane Nodes
Thousands of HVs &
Region Control Plane Nodes
Tens of Thousands of HVs &
Control Plane Nodes
Global
Cloud
Region Region
Cell Cell Cell
HV HV HV HV HV HV
Cell Cell
Region
8
Maintain at Cloud Scale
Operating OpenStack @ Rackspace
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Code
Package
Deploy
Verify
9
Updating the Control Plane
Operating OpenStack @ Rackspace
During a control plane
update, customer
instances on the
hypervisor stay up and
running.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
10
Code Iterations & Releases
Operating OpenStack @ Rackspace
6-Month OpenStack Release Cycle
60 Days
60 Days
60 Days
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Benefits and Challenges of Trunk Deploys
Operating OpenStack @ Rackspace
11
Why We Do It (Benefits)
• Issue Resolution
– Early detection of issues and conflicts
– Shorter feedback loop within the
community
– Faster resolution of issues
• Early Feature Delivery
– Smaller, incremental periodic releases
– More stable release candidates for the
community at end of cycle
Why It’s Hard (Challenges)
• Code Management
– Merge conflicts with local patches
– Disruptive DB migrations
– Service restarts
– Temporary version skew
• Testing
– Devstack-based testing vs testing at
scale
– Rework when issues found in RAX deploy
pipeline
• Process
– CI/CD vs 6-month Releases
– Time to merge upstream patches
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• per-service venv
• .tar of service
venvs + configs
Package
• seed .torrent
• distribute configs
• verify completion
Distribute
• switch version
• sync databases
• apply configs
• verify completion
Execute
12
Package and Deploy
Operating OpenStack @ Rackspace
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Deploy and Verify
Operating OpenStack @ Rackspace
• pre-code
check-in
validation
Dev
• smoke tests
• unit tests
Integration
• functional tests
• integration
tests
QA
• regression
tests
• build tests
Pre-Prod
• smoke tests
• build tests
Production
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Jenkins
Ansible
Puppet
• Execution
• Orchestration
• Configuration
Management
14
Tools to Maintain the Cloud
Operating OpenStack @ Rackspace
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
15
Tools to Maintain the Cloud
Operating OpenStack @ Rackspace
• Puppet
– Configuration management for all control plane nodes
– YAML-defined fact files for config variables
– Masterless configuration to prevent bottlenecks at scale
• Ansible
– Playbooks for orchestration of OpenStack service bootstrap and updates
– Playbooks for orchestration of out-of-iteration hotpatches and changes
• Jenkins
– Scripts called to create virtual environment (venv) packages
– Jobs configured for automating the pull and merge of upstream code
– QE test automation triggered after deployments
– Ansible playbooks invoked from jenkins to reduce human error
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• The hybrid cloud concept will enable individuals to access the
infrastructure they need when they need it.
• One part of the hybrid cloud portfolio at Rackspace is the public cloud,
powered by OpenStack.
• OpenStack is an open source cloud operating system with distinct
services that work together to enable the creation of cloud servers.
• Rackspace uses a variety of tools and processes to maintain
OpenStack services throughout the OpenStack release cycle.
16
Final Thoughts
Operating OpenStack @ Rackspace
Twitter: @rainyamosher | Email: [email protected]
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Backup Slides
Operating OpenStack at Rackspace
17
Many of these backup slides were first presented on 4/16/2013 during the
OpenStack Summit session “Deploying from OpenStack Trunk” and are
included here for reference.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
18
OpenStack Release Cycle
Operating OpenStack @ Rackspace
6-Month OpenStack Release Cycle
new features accepted feature freeze
Planning
Community designs,
discusses, & targets
release cycle
Implementation
Community makes
changes to OpenStack
code & creates new
functionality
Pre-Release
Community focuses
on bug fixes, docs,
& testing
Maintenance
Security vulnerability fixes are supported for the last three Releases
Final
Release
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
19
Rackspace Iteration Cycle
Operating OpenStack @ Rackspace
Choose Release
Branch
Pull upstream code,
merge with internal
patches, and deploy
to CI environment
for validation
Release Branch
Integration
Increase test coverage on
release branch candidate
to include full end-to-end
integration testing with
Rackspace services
Approve Release
Once testing
passes, release is
approved and
production deploy is
scheduled
60-Day Rackspace Iteration Cycle
code may change release branch frozen
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
20
Merge and Branch Strategy
Operating OpenStack @ Rackspace
• Rackspace Development branch is
about 40 patches on top of
OpenStack trunk for internal service
compatability
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
21
Scale of Deploy Pipeline
Learning to Scale OpenStack
1,000s of Nodes 100s of Nodes 10s of Nodes DevStack
Dev
Integration
& QA
PreProd Production
22
RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218
US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM
RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM
doc_236095834.pptx