Category Archives: IBM

IBM SoftLayer IaaS – notes from 2 day training class in NYC

I attended SoftLayer training in NYC and wrote up a few pages of notes. I really like the idea of building IaaS systems via web control panels and APIs, and SoftLayer delivers on this.


  • 21k customers in 140 countries
  • 15 data centers, 18 network points of presence (PoPs)
  • Mix and match of virtual (diverse set of hypervisors) and bare metal servers, all managed via web control panel and/or API
  • Deployment in real time with high degree of automation.
  • Some customers build a hybrid solution using SoftLayer in addition to their own datacenter. Connect via VPN or leased line.

Server Architecture

  • While most cloud providers offer only virtualized resources on shared infrastructure, SoftLayer offers the option of bare metal and/or virtualization, and the option of shared and/or dedicated infrastructure.
  • Redundancy in some cases stops at the rack, not the server. For example, multiple power supplies for the rack not for each server in the rack
  • Server options
    • Multi-tenant (you don’t know who/what else is running on the same resources as you)
      • Virtual (public node)
        • Managed Citrix Xen hypervisor
        • Monthly/Hourly billing
        • Up to 16 cores
        • Local storage or SAN
        • Free 5 TB outbound data transfer if choose monthly billing
        • 15 minute provisioning
      • Single tenant (all resources dedicated to single customer, aka “private cloud”)
        • Bare metal
          • Optional (unmanaged) hypervisor, such as Citrix Xen, VMWare, Hyper-V, Parallels
          • Monthly billing. In some instances can do hourly billing
          • Free 20 TB outbound bandwidth per month
          • Optional private network, private rack
          • Options on CPUs, up to 36 internal drives (build your own NAS), NVIDIA Tesla GPU
          • 2-4 hour provisioning. That’s the time it takes for the machine to become visible to the customer. Additional time needed to apply operating system and applications.
        • Virtual (private node)
          • Pretty much the same as Multi-tenant virtual except that you have dedicated hardware.
          • You can install as many virtual machines as you want on your hardware.
    • OK for customer to deploy their own software appliances, but there is no option to ever deploy your own hardware
    • Image Templates
      • Software/configuration of a physical or virtual space
      • Apply to a machine to create a runtime environment
      • Two types of image templates
        • Standard
          • Virtual machine only
          • Any operating system
          • Citrix Xen only
        • Flex
          • Both physical and virtual machines
          • Red Hat (RHEL) and Windows only
          • All hypervisors


  • Three networks
    • Public (2 NICs, both usable rather than just redundancy)
      • Bare metal: 20 TB outbound bandwidth per month
      • Virtual: 5 TB outbound bandwidth per month. Can be pooled if some servers aren’t publically exposed
    • Private (2 NICs, both usable rather than just redundancy)
      • No limitations on bandwidth. Great for backups across multiple datacenters
      • Private VLANs can include servers in multiple datacenters. A server can connect (span) to multiple VLANs
    • Management/Admin (1 NICs)
  • SoftLayer SLA: “reasonable efforts to provide 100% service”
  • VPN
    • tunnels: SSL, PPTP, IPSec
    • Recommends managing with FortiGate or Vyatta appliances
  • SoftLayer Looking Glass: Test latency between your datacenter and SoftLayer, or between resources within or across SoftLayer datacenters
  • Content Delivery Network
  • Load Balancing
  •  Firewalls
    • Fortinet FortiGate 3000 series
    • Shared hardware
      • Multi-tenant
      • Managed through Customer Portal & APIs. No console access because it’s shared hardware.
      • Configured to protect a single server
    • Dedicated hardware
      • Same as above, but single-tenant, yet still no console access.
      • Configured to protect a single server or an entire VLAN
    • Dedicated appliance
      • Same as dedicated hardware, but provides access to console and native tools. This gives the customer more capabilities.
  • Gateway Appliance
    •  Vyatta
      • Applies to any portion of, or entire customer infrastructure at SoftLayer
      • Used forGateway Appliance
        • IPSec VPN tunnels
        • NAT
        • Firewall
        • Router
      • Configured by console or Vyatta gui via VPN. No SoftLayer Customer Portal or API
  • DNS Options
    • Customer uses their own DNS that’s external to SoftLayer
    • Customer uses SoftLayer’s DNS, which is redundant across datacenters
    • Customer uses 3rdparty DNS
    • Customer runs their own DNS hosted on their own machines within SoftLayer


  • Much easier to deploy/configure security via the SoftLayer Customer Portal than in a traditional datacenter. One common source of vulnerabilities is incomplete or incorrect security deployments, so an easier to use method would suggest that it’s easier to create a secure system.
  • Offerings
    • McAfee (Windows) anti-virus
    • DDoS – detect and isolate (take off line) machines that are under attack, but does not have service to remediate the threat
      • Cisco Guard DDoS protection
      • Arbor Peakflow traffic analysis
      • Arbor ATLAS Global Traffic Analyzer
    • Servers local to datacenter for Windows and Red Hat updates
    • IDS/IPS protection
      • Nessus vulnerability assessment and reporting
      • McAfee host intrusion protection
    • FortiGate firewalls
    • US Gov’t standards
      • Drive wiping using same tools as Dept of Defense (DoD)
      • SP800-53 US Gov’t standard
      • Federal Information Security Management Act (FISMA).
      • FedRAMP datacenters
      • Health Insurance Portability and Accountability Act (HIPPA). Will sign agreement with customer.
    • Two factor authentication
      • Symantec identity protection
      • Windows Azure Mult-Factor
    • VPN
      • Client site SSL or PPTP, and Site to site IPSec
  • Datacenters are
    • Service Organization Control (SOC) 2 certified
    • Payment Card Industry Security Standard (PCI-DSS) for bare metal and single-tenant virtual. Not recommended for multi-tenant.
    • Tier 3
      • 99.982% availability (translates to < 1.6 hours/year)
      • Multiple power/cooling
      • N+1 fault tolerant
      • Can sustain 72 hour power outage
    • Physical security. All items mentioned are good, but seemed typical of other datacenters I’ve been to or learned about.
    • Cloud Security Alliance (CSA) self-assessment, but not yet certified


Managed services

  • Backup plans
  • Security plans, patching, server hardening
  • Monitoring
  • DBA
  • Change Management


  • Implemented using SOAP and XML-RPC
  • Available as Representational State Transfer (REST)
  • Supports a wide range of languages
  • 264 services (20 of which are high level) comprising a total of 3,421 API calls
  • Can be used to up-scale and down-scale an implementation in an automated manner. There’s a new package for this called OnScale. Not sure at what level this compares or competes with Pure Applications on SoftLayer
  • Can be used to create a custom branded Customer Portal for reselling services

Compared to other cloud providers

  • A lot of marketing hype, although Gartner quadrant wasn’t at all kind to SoftLayer
  • Compared to Amazon AWS showed as higher performance and availability at lower cost, but used bare metal for the comparison. Didn’t show whether SoftLayer virtual is comparable to AWS, although in theory SoftLayer would cost less.
  • Catalyst: incubator to help small companies with infrastructure costs

IBM PureApplications for Hybrid IaaS Cloud

IBM PureApplications provides on-premise cloud. #PureApp for SoftLayer provides off-premises cloud solutions. @Prolifics

Video includes clip from my manager @Prolifics, Mike Hastie.

Small Data can also be Big Data

In 2001 The Meta Group (later acquired by Gartner) defined “Big Data” using the “3 Vs”

  1. Volume
    • Amount of data
    • By one estimate in 2013, 90% of all digital data has been created since 2011
    • The Square Kilometer Array Telescope (  will generate approx 1 exabyte of data per day
    • IDC defines Big Data projects as having at least 100 terabytes, which is naïve since this would preclude the vast majority of organizations from justifying a Big Data project.
  2. Velocity
    • Frequency of data in and out
    • Not just daily batch uploads
    • Google stores meta data about searches that people perform – approx 2.5 million per minute
  3. Variety
    • Range of data types
    • Range of data sources
    • Most data created is unstructured. Some is semi-structured (spreadsheets?). Very little is structured (forms).
    • An example of “variety” is a bakery which has many kinds of bread

Another article defines an additional 4 Vs

  1. Veracity
    • Processes to ensure that the data is correct. This seems intuitive for all database, but the scale of Big Data also scales the impact of data inconsistencies.
  1. Variability
    • Data that changes meaning based on context (of other data, or of time). This requires far more complexity of analysis than can be done using a traditional SQL relational database system.
    • An example of “variability” is a bakery in which the sourdough bread is just a little bit different every few days on an unpredictable schedule. The variety is the same but attributes of it are variable. If one could predict on which days one could expect specific tastes, then it would simply be variety.
  1. Visualization
    • This becomes difficult to do only because of the previous five Vs. With simple data, the difficulty of visualization scales linearly. With Big Data, the difficulty of visualization is a factor of each of the 7 Vs.
  1. Value
    • Ability for analysis of the data to be assigned as worth a lot of money. I don’t agree that this should be a V at all, since it is dependant upon or created by dependencies for any of the other 6 Vs. Can’t simple data also have great value? Just because data is Big, doesn’t imply that it is possible to extract value.

This is in contrast to IBM’s definition of Small Data, which seems to be simply “not 3 Vs”

  1. Low Volume
    • Big Data projects are viable even if the volume is only a few gigabytes. In this case you have only a few nodes in your cluster.  There’s no shame in having a Big Data infrastructure that isn’t thousands of nodes.
  2. Batch Velocity
    • Sometimes the source can generate only in batches, not real time, so should not disqualify the project if criteria for other Vs are met.
  3. Structured Data
    • Web Server log files are often an input into a Big Data system yet are not structured. At best they are semi-structured.


Article on IBM DeveloperWorks

Open Source Big Data for the Impatient, Part 1: Hadoop tutorial: Hello World with Java, Pig, Hive, Flume, Fuse, Oozie, and Sqoop with Informix, DB2, and MySQL