Category Archives: AWS

IBM SoftLayer IaaS – notes from 2 day training class in NYC

I attended SoftLayer training in NYC and wrote up a few pages of notes. I really like the idea of building IaaS systems via web control panels and APIs, and SoftLayer delivers on this.


  • 21k customers in 140 countries
  • 15 data centers, 18 network points of presence (PoPs)
  • Mix and match of virtual (diverse set of hypervisors) and bare metal servers, all managed via web control panel and/or API
  • Deployment in real time with high degree of automation.
  • Some customers build a hybrid solution using SoftLayer in addition to their own datacenter. Connect via VPN or leased line.

Server Architecture

  • While most cloud providers offer only virtualized resources on shared infrastructure, SoftLayer offers the option of bare metal and/or virtualization, and the option of shared and/or dedicated infrastructure.
  • Redundancy in some cases stops at the rack, not the server. For example, multiple power supplies for the rack not for each server in the rack
  • Server options
    • Multi-tenant (you don’t know who/what else is running on the same resources as you)
      • Virtual (public node)
        • Managed Citrix Xen hypervisor
        • Monthly/Hourly billing
        • Up to 16 cores
        • Local storage or SAN
        • Free 5 TB outbound data transfer if choose monthly billing
        • 15 minute provisioning
      • Single tenant (all resources dedicated to single customer, aka “private cloud”)
        • Bare metal
          • Optional (unmanaged) hypervisor, such as Citrix Xen, VMWare, Hyper-V, Parallels
          • Monthly billing. In some instances can do hourly billing
          • Free 20 TB outbound bandwidth per month
          • Optional private network, private rack
          • Options on CPUs, up to 36 internal drives (build your own NAS), NVIDIA Tesla GPU
          • 2-4 hour provisioning. That’s the time it takes for the machine to become visible to the customer. Additional time needed to apply operating system and applications.
        • Virtual (private node)
          • Pretty much the same as Multi-tenant virtual except that you have dedicated hardware.
          • You can install as many virtual machines as you want on your hardware.
    • OK for customer to deploy their own software appliances, but there is no option to ever deploy your own hardware
    • Image Templates
      • Software/configuration of a physical or virtual space
      • Apply to a machine to create a runtime environment
      • Two types of image templates
        • Standard
          • Virtual machine only
          • Any operating system
          • Citrix Xen only
        • Flex
          • Both physical and virtual machines
          • Red Hat (RHEL) and Windows only
          • All hypervisors


  • Three networks
    • Public (2 NICs, both usable rather than just redundancy)
      • Bare metal: 20 TB outbound bandwidth per month
      • Virtual: 5 TB outbound bandwidth per month. Can be pooled if some servers aren’t publically exposed
    • Private (2 NICs, both usable rather than just redundancy)
      • No limitations on bandwidth. Great for backups across multiple datacenters
      • Private VLANs can include servers in multiple datacenters. A server can connect (span) to multiple VLANs
    • Management/Admin (1 NICs)
  • SoftLayer SLA: “reasonable efforts to provide 100% service”
  • VPN
    • tunnels: SSL, PPTP, IPSec
    • Recommends managing with FortiGate or Vyatta appliances
  • SoftLayer Looking Glass: Test latency between your datacenter and SoftLayer, or between resources within or across SoftLayer datacenters
  • Content Delivery Network
  • Load Balancing
  •  Firewalls
    • Fortinet FortiGate 3000 series
    • Shared hardware
      • Multi-tenant
      • Managed through Customer Portal & APIs. No console access because it’s shared hardware.
      • Configured to protect a single server
    • Dedicated hardware
      • Same as above, but single-tenant, yet still no console access.
      • Configured to protect a single server or an entire VLAN
    • Dedicated appliance
      • Same as dedicated hardware, but provides access to console and native tools. This gives the customer more capabilities.
  • Gateway Appliance
    •  Vyatta
      • Applies to any portion of, or entire customer infrastructure at SoftLayer
      • Used forGateway Appliance
        • IPSec VPN tunnels
        • NAT
        • Firewall
        • Router
      • Configured by console or Vyatta gui via VPN. No SoftLayer Customer Portal or API
  • DNS Options
    • Customer uses their own DNS that’s external to SoftLayer
    • Customer uses SoftLayer’s DNS, which is redundant across datacenters
    • Customer uses 3rdparty DNS
    • Customer runs their own DNS hosted on their own machines within SoftLayer


  • Much easier to deploy/configure security via the SoftLayer Customer Portal than in a traditional datacenter. One common source of vulnerabilities is incomplete or incorrect security deployments, so an easier to use method would suggest that it’s easier to create a secure system.
  • Offerings
    • McAfee (Windows) anti-virus
    • DDoS – detect and isolate (take off line) machines that are under attack, but does not have service to remediate the threat
      • Cisco Guard DDoS protection
      • Arbor Peakflow traffic analysis
      • Arbor ATLAS Global Traffic Analyzer
    • Servers local to datacenter for Windows and Red Hat updates
    • IDS/IPS protection
      • Nessus vulnerability assessment and reporting
      • McAfee host intrusion protection
    • FortiGate firewalls
    • US Gov’t standards
      • Drive wiping using same tools as Dept of Defense (DoD)
      • SP800-53 US Gov’t standard
      • Federal Information Security Management Act (FISMA).
      • FedRAMP datacenters
      • Health Insurance Portability and Accountability Act (HIPPA). Will sign agreement with customer.
    • Two factor authentication
      • Symantec identity protection
      • Windows Azure Mult-Factor
    • VPN
      • Client site SSL or PPTP, and Site to site IPSec
  • Datacenters are
    • Service Organization Control (SOC) 2 certified
    • Payment Card Industry Security Standard (PCI-DSS) for bare metal and single-tenant virtual. Not recommended for multi-tenant.
    • Tier 3
      • 99.982% availability (translates to < 1.6 hours/year)
      • Multiple power/cooling
      • N+1 fault tolerant
      • Can sustain 72 hour power outage
    • Physical security. All items mentioned are good, but seemed typical of other datacenters I’ve been to or learned about.
    • Cloud Security Alliance (CSA) self-assessment, but not yet certified


Managed services

  • Backup plans
  • Security plans, patching, server hardening
  • Monitoring
  • DBA
  • Change Management


  • Implemented using SOAP and XML-RPC
  • Available as Representational State Transfer (REST)
  • Supports a wide range of languages
  • 264 services (20 of which are high level) comprising a total of 3,421 API calls
  • Can be used to up-scale and down-scale an implementation in an automated manner. There’s a new package for this called OnScale. Not sure at what level this compares or competes with Pure Applications on SoftLayer
  • Can be used to create a custom branded Customer Portal for reselling services

Compared to other cloud providers

  • A lot of marketing hype, although Gartner quadrant wasn’t at all kind to SoftLayer
  • Compared to Amazon AWS showed as higher performance and availability at lower cost, but used bare metal for the comparison. Didn’t show whether SoftLayer virtual is comparable to AWS, although in theory SoftLayer would cost less.
  • Catalyst: incubator to help small companies with infrastructure costs

Xplenty offers HaaS in AWS

Xplenty offers Hadoop as a Service for Amazon Web Services in all AWS global regions. This HaaS offering promises a “coding free design environment”, of course in additional to AWS hardware free environment.


HaaS Provider Qubole Now Runs on Google Compute Engine (GCE)

I’m starting to see applications ported from AWS to GCE, but not sure about the justifications for running production systems on GCE. Maybe price?


Spotify Embraces Hortonworks, Dumps Cloudera

Cloudera had appeared to be the defacto standard of Hadoop distributions, but Hortonworks has scored big in this deal. Spotify has a 690 node Cloudera cluster that it will be moving to a Hortonworks cluster (undisclosed size). Apparently it’s the new Hive implementation that makes Hortonworks so attractive.

When Spotify launched in 2008 it had a 30 node cluster hosted on Amazon’s AWS, then switched to an on-premises 60 node cluster that grew to 690 nodes. The cluster currently contains 4 petabytes of data which grows by 200 gigabytes per day.

Spotify has a 12 person Hadoop team and uses a Python (not Java) framework for batch processing.


Use Cases for Hadoop

The Apache “Powered by Hadoop” page lists a long list of companies that use Hadoop. Some only list the company name. Others have a sentence or two about what they’re using Hadoop for. And some, like LinkedIn, list the specs of the hardware that they have Hadoop running on. There’s also a link to a great article about how the NYTimes used Hadoop  (in 2007!!!) running on Amazon’s AWS cloud to generate PDFs of 11 million articles in 24 hours running on 100 machines. One of the things I find interesting about the NYTimes use case is that they used Hadoop for a one time batch process. A lot what we read about Hadoop assumes that the use case is an ongoing, maybe multi-year application.


Cheap Hardware

In theory, a big data cluster uses low cost commodity hardware (2 CPUs, 6-12 drives, 32 GB RAM). By clustering many cheap machines, high performance can be achieved at a low cost, along with high reliability due to decentralization.

There is little benefit to running Hapdoop nodes in a virtualized environment (e.g VMWare), since when the node is active (batch processing) it may be pushing RAM and CPU utilization to its limits. This is in contrast to an application or database server which has idle and bursts, but generally has constant utilization at some medium level. What is of greater benefit is a cloud implementation (e.g. Amazon Elastic Cloud) in which one can scale from a few nodes to hundreds or thousands of nodes in real time as the batch cycles through its process.

Unlike a traditional n-tier architecture, Hadoop combines compute & storage on the same box. In contrast, an Oracle cluster would typically store its databases on a SAN, and application logic would reside on yet another set of application servers which probably do not utilize their inexpensive internal drives for application specific tasks.

A Hadoop cluster is linearly scalable, up to 4000 nodes and dozens of petabytes of data.

In a traditional db cluster (such as Oracle RAC), the architecture of the cluster should be designed with knowledge of the schema and volume (input and retrieval) of the data. WIth Hadoop, scalability is, at worst, linear. Using a cloud architecture, additional Hadoop nodes can be provisioned on the fly as node utilization increases or decreases.