My personal laptop, reasonably high-end for the time, had a 5400 RPM spinning disk and 2 GB of RAM. SSDs were exotic, even on servers. Nowadays, you can get bare metal servers with 512gb-1tb of RAM, 2x multi-core CPUs and terabytes of fast SSD storage for a price that is perfectly reasonable for even small companies. Similarly, you can easily and cheaply launch fleets of high-spec virtual servers with providers like Amazon Web Services and DigitalOcean at minutes’ notice.

In many ways, it seems to me that we are often basing architectural decisions on imagined constraints. In my experience, a decision to embrace a microservices architecture should not follow primarily from concerns about scalability.

Typically the burden and overhead of managing several services across several environments (development, testing, QA, production, etc) is a huge multiple of that of managing a more monolithic codebase. Furthermore, scaling a monolithic application, within most practical bounds, is actually often simpler and cheaper than scaling a more distributed app.

From a technical perspective (speaking in this instance of web apps) a monolithic application can scale very naturally. The application tier can scale horizontally to an almost infinite degree by adding more application servers. Particularly high-traffic pages with largely static content can easily be placed behind a reverse proxy cache like Varnish (or a commercially-hosted sibling like Fastly). High-traffic pages with more dynamic content can still have their performance dramatically improved with strategies like fragment caching (using a memory store like Redis or Memcached). Relational databases can scale to immense capacity either in hosted/managed forms (such as Amazon RDS) or hosted on your own hardware. Master-Slave replication schemes can allow database reads to scale in a horizontal manner similar to scaling the application tier. Only extremely write-heavy apps present any significant challenges in this area, and even these scenarios now have a multitude of purpose-built solutions such as Cassandra and Citus (this is also not something that will be overcome any more easily with a microservices solution).

So when should you adopt microservices solutions? To me there are two especially compelling scenarios. One is what I would call the “service bridge” scenario. This would be where you have a niche feature that has a significantly different traffic profile to your larger app and, more importantly, would introduce extremely awkward dependencies to your application tier.

A good example of this might be something like IP geolocation, which could require data sets of hundreds of megabytes or more (assuming something like the Maxmind’s binary data files) that you may not want to shoehorn into your primary application (so as not to bloat your application server). Such a niche dependency might be better implemented as a microservice (though I would argue you would probably be better off delegating to a hosted provider with an API).

Microservices architectures are also well-suited in circumstances where you have a very large organization with many domain-focused teams that would benefit from a very high degree of autonomy. One of the organizations most visibly advocating for and implementing service oriented architectures early on was Amazon (as  wonderfully documented by Steve Yegge in his famous Google Platforms Rant [archive link]). It’s arguable that this vision of service oriented architecture (SOA) is more along the lines of having multiple large, monolithic applications with distinct teams and some data shared, rather than the common understanding of microservices (which is more akin to single applications composed of several small services).

When adopting microservices, be mindful of the unique challenges of the architecture, and have a plan to address them. These should not be incidental concerns but a primary focus from the outset if your team is to thrive. Things such as bootstrapping the development environment and having cohesive QA and versioning practices can be challenging with a microservices architecture. So too can logging and tracing. Many (especially in the context of smaller organizations) take an ad-hoc approach to these issues because they can still manage to make the system function, but oversights of this nature can become serious liabilities at scale.

The critical thing that I hope to convey is that microservices should not be adopted as a default solution for the problem of scaling an application. They can be a great fit for scaling teams and organizations, as well as for wrapping up functionality that it is particularly impractical to fit within your primary application’s deployment. The matter of scaling an application can be addressed extremely effectively with a monolithic codebase and traditional horizontal scale-out methods.

The post On microservices and distributed architectures appeared first on Superset Blog.

For our own application, we have a few geographically-distributed Nginx hosts functioning as “dumb” edge servers in a CDN. These Nginx hosts proxy back to object storage to serve up large media files to users. For each file access we want to record analytics information such as IP, user agent, and a client identifier (via the Nginx userd module, to differentiate unique users to the extent that it is possible).

For the purposes of this article we will go through the manual steps of configuration (on Ubuntu Server) though you should absolutely be using a configuration management tool such as Ansible to automate these steps in a production, as we have.

The dependencies for this logging configuration are nginx-extras (which includes the userid module for analytics) and rsyslog-pgsql (the package for the Rsyslog Postgres output module, which is not part of the default Rsyslog install). You can install these with apt (either manually or via Ansible’s apt module):

sudo apt-get install nginx-extras rsyslog-pgsql

Ubuntu should have a top-level Rsyslog configuration file at /etc/rsyslog.conf which should end with the line:

# ...
$IncludeConfig /etc/rsyslog.d/*.conf

This instructs the Rsyslog daemon to pull in any configuration files contained in the directory /etc/rsyslog.d when it loads. We will use this to set up a special configuration to pull-in and forward our formatted nginx logs momentarily. First, let’s configure nginx to log a json payload to a unique log file.

Ubuntu’s standard nginx configuration pulls in per-site config files from the /etc/nginx/sites-available (and assumes you have sym-linked the configurations for sites you want to go live to /etc/nginx/sites-enabled/). For this example, we’ll assume a configuration for mysite.com in /etc/nginx/sites-available/mysite.com.conf:

# in /etc/nginx/sites-available/yoursite.com:

log_format json_combined '{"time_local": "$time_local", '
   '"path": "$request_uri", '   
   '"ip": "$remote_addr", '
   '"time": "$time_iso8601", '
   '"user_agent": "$http_user_agent", '
   '"user_id_got": "$uid_got", '
   '"user_id_set": "$uid_set", '
   '"remote_user": "$remote_user", '
   '"request": "$request", '
   '"status": "$status", '
   '"body_bytes_sent": "$body_bytes_sent", '
   '"request_time": "$request_time", '
   '"http_referrer": "$http_referer" }';

server {    
  listen 80;
  # + SSL configuration...

  # Optional: Nginx userid module, useful for analytics.
  # (see http://nginx.org/en/docs/http/ngx_http_userid_module.html)
  userid on;
  userid_name uid;
  userid_expires 365d;

  server_name yoursite.com;
  # any additional server-level configuration such as site root, etc...
 
  location / {
    access_log /var/log/nginx/yoursite.com/access.log json_combined;
  } 
  # You will probably want to add some gzip, cache, etc standard header rules for performance...
}

And load the Nginx configuration with:

sudo service nginx reload

We now have Nginx writing a one-off access log with a json-formatted payload to /var/log/nginx/yoursite.com/access.log. We can configure Rsyslog to read this log file using the imfile module. In the past imfile defaulted to a polling mode but today defaults to filesystem events and is very performant and effectively real-time. With the imfile module reading the log file on Rsyslog’s behalf, we can then forward the log file data to Postgres using the Rsyslog ompgsql (PostgreSQL Database Output) module. The combined configuration is as follows:

# Load the imfile input module
module(load="imfile") # Load the imfile input module

input(type="imfile"
      File="/var/log/nginx/yoursite.com/access.log"
      Tag="yoursite:")

# Load the ompgsql output module
module(load="ompgsql")

# Define a template for row insertion of your data.
# The template below assumes you have a table called
# "access_log" and are inserting columns named 
# "log_line" (with the log payload) and "created_at" (with the timestamp).
template(name="sql-syslog" type="list" option.sql="on") {
  constant(value="INSERT INTO access_log (log_line, created_at) values ('")
  property(name="msg")
  constant(value="','")
  property(name="timereported" dateformat="pgsql" date.inUTC="on")
  constant(value="')")
}      

# The output "action". This line instructs rsyslog
# to check if the log line is tagged "yoursite:" (a tag
# which we set with the imfile module configuration above)
# and if so to use the sql-syslog template we defined
# above to insert it into Postgres.
if( $syslogtag == 'yoursite:')  then {
  action(type="ompgsql" server="{{ postgres_host }}"
        user="{{ postgres_user }}" 
        pass="{{ postgres_password }}"
        db="{{ postgres_db_name }}"
        template="sql-syslog"
        queue.type="linkedList")
}

You will want to name this file something like /etc/rsyslog.d/51-yoursite.conf, since Rsyslog loads config files in alphabetical order and on Ubuntu has a default configuration file in the same directory called 50-default.conf. It probably goes without saying but the ompgsql “action” line in the configuration above is using mock templatized credentials (I can recommend Ansible Vault for managing/templating credentials such as these in production). I should also note that as Rsyslog is a very long-lived project, it supports several different configuration file formats. The example above is using the “advanced” (a.k.a. “RainerScript”) format because I find this to be by far the most readable. Once you have saved the above log file, you will need to restart the Rsyslog daemon for it to take effect:

sudo service rsyslog restart

The above configuration should be pretty performant as the “linkedList” queue.type argument supplied to the ompgsql action is instructing Rsyslog to buffer/batch its writes to Postgres. You can read about the performance tweaking that is available for ompgsql in an excellent article, “Handling a massive syslog database insert rate with Rsyslog“, which was written by Rainer Gerhards himself (primary author of Rsyslog).

The post High-performance logging from Nginx to Postgres with Rsyslog appeared first on Superset Blog.

I see many teams break out their applications into services prematurely, with immense consequence in terms of cognitive overhead and loss of velocity. Typically the decision is a consequence of a perception that they have hit a performance ceiling, when in fact they are still on some relatively weakly-provisioned PaaS.

I want to do a detailed value comparison to aid others in making informed infrastructure decisions.

For the purposes of this comparison, we will consider two scenarios: what I would call a mid/moderate-scale app, with 60GB of data in persistence storage and a requirement to handle 200 requests/second, and what I will call a higher-scale app, with 1 TB of storage in use and a requirement to handle 3,000 requests/second.

We will also assume a relatively monolithic 12-factor application architecture, with an application tier, an application caching tier and a persistence tier.

There is necessarily a lot of variability in the performance profile and resource usage of applications. For the purpose of this comparison, we will assume a Ruby on Rails application running atop the Passenger web application server. We will assume each application process consumes 300MB, and we will use at most 88% of the available memory on each application server (which means we would fit 3 worker processes on an application server with 1024MB of RAM). The mid-scale application will therefore require 10 application processes (20x reqs/second/process x 10 processes = 200 requests/second), and the high-scale application will require 150 application processes.

We will assume the application is tuned well enough to average 50ms server times. This would mean that we can assume each server process we have active can roughly accommodate 20 reqs/second.

This makes the cost estimates in this comparison optimistic, as it does not account for sub-optimal request routing/load-balancing (particularly relevant to Heroku) nor the fact that very few applications will have a performance profile that reliably keeps requests at a 50ms/second average server time.

Category 1: PaaS

Note: based on our resource utilization profile outlined above, we assume that a Standard 2x Dyno can accommodate 3 worker processes, a Performance M Dyno can accommodate 7 processes (in 88% = 2.2GB of its total 2.5GB of memory) and a Performance L Dyno can accommodate 41 processes.

Category 2: VPS

For our VPS survey we will have a look at two of the leading VPS providers: Linode and DigitalOcean. Linode’s pricing is quite a lot better than DigitalOcean’s, but they have a public history of DDoS episodes that have taken major sites offline and an arguably less rich API and admin experience. I still consider Linode highly viable and think it is worth taking a close look if at both providers’ offerings if you are considering moving your infrastructure to VPS.

Linode

DigitalOcean

Category 3: Cloud

For Cloud, we will consider configurations on Amazon Web Services. We will keep things roughly aligned with our VPS specs. It is important to note that opting for Reserved pricing can realistically lower your AWS costs as much as 40-45%, but you must have a rough idea of what your resource utilization will be at least one year forward, so we will consider both reserved and On-Demand pricing here. We have used 40% as a rough benchmark since the precise calculations will be unique to your specific service selection and whether you are prepaying or not.

* note: we have not been very specific in calculations concerning block storage (EBS) costs at the application tier. The requirements at this tier (if you are using block storage at all) are likely to be insubstantial (~50GB). Opting for provisioned IOPS may raise your costs into the low $100s. If relying on General Purpose SSD + ephemeral storage at the instance level you will likely pay on the order of $20 or less as assumed here.

Category 4: Dedicated

It is not wholly straightforward to price a dedicated configuration as dedicated servers can be so generously provisioned that at moderate scale (such as our “mid-scale” application) it often makes sense to run all services on a single host. “High-availability” multi-host configurations are also possible and appropriate at larger scales or in contexts that require high availability. We will assume a high-availability, multi-host configuration for our “high-scale” application and a single-host configuration for the “mid-scale” application. We will consider both the budget host OVH (good value but spotty reputation for support and even uptime) and the slightly costlier Liquid Web (US-based managed host).

OVH

OVH offers good value dedicated hosting in several datacenters around the world. Their BHS/Quebec location is most relevant for American companies. There are some big caveat to OVH’s dedicated offering and it may not suit all applications. They have a history of leaving customers relatively on their own to sort out everything short of hardware failure. I also once firsthand experienced an episode of hours-long downtime due to a road accident that severed the datacenter’s fiber line (it is worrying that there was neither better fortification nor sufficient redundancy to keep hosts online). This one episode is the only major one I experienced however and it is well known that AWS services also go down, sometimes for similar periods of time, and sometimes even across all availability zones (such as memorably occurred with S3 in 2016).

Liquid Web

Liquid Web offers managed hosting of dedicated servers. I include them in this comparison to contrast OVH since OVH is hands-off in a degree that may not align with the requirements of some businesses or teams. You pay a premium for a hands-on managed hosting solution but value is still good despite this.

The Wildcard: Colocation

At a certain scale you may find your best fit to be none of the discussed solutions but rather colocation. With colocation you rent space in a data center (typically by the quarter, half or full-cabinet) and pay for power and bandwidth, supplying your own hardware. This means paying hardware costs upfront and then paying comparatively quite a lot less in your ongoing hosting costs. There are also potential accounting advantages to be realized with colocation (as you own the hardware, you can write down its depreciation). You will need to have either a team or team members who are experienced in managing physical servers, and colocation is likely not a good fit for all but very high-scale projects. If you want to read a good overview of a colocation deployment in practice, I recommend having a look at Nick Carver’s article about Stack Overflow’s infrastructure.

Conclusion

I hope this article provided a rough overview of the comparative costs of various paths you might follow with your infrastructure. Be aware that it is neither exhaustive nor conclusive. There is a lot of variability when it comes to architectures and needs, and the numbers used should be considered only a rough approximation of value. It should also be understood that in order to utilize non-PaaS infrastructure solutions well, ergonomic tooling and processes (for things such as bootstrapping new environments, database backup and restore, deployment, etc.) will be required. We have posted elsewhere on this subject (see: “How to mess up DevOps: working at the wrong level of abstraction“) so please refer to those articles or elsewhere on the web for info about infrastructure management and DevOps tooling.

We provide a matrix below to summarize the data from this article:

The post Dedicated vs. Cloud vs. VPS vs. PaaS – a value comparison appeared first on Superset Blog.

A tech company gets its product off the ground with a small handful of developers and a user-friendly fully hosted Platform-as-a-Service (PaaS) solution like Heroku.

The company’s product is a success. A huge one! The company raises money, they scale the team, they iterate. One thing that doesn’t change is the PaaS. It’s working for them. Maybe not as well as they’d like but well enough to keep up with the roadmap.

At some point, costs get way out of hand. The once $1k/month bill has exploded to $40k/month. On top of this, developers are sick of hacking around arbitrary constraints of the PaaS. They learn of the dramatically better performance they can achieve at lower cost if they take greater ownership of their infrastructure.

They engage consultants. Often dubious ones.

The consultants come in with fantastic promises and build with buzzword-y tools. Projects drag on for months. Maybe years. If anything does see the light of day from the consultants’ efforts, it is so bespoke and complicated that simple tasks require several arcane commands that developers can never keep in their heads.

In most cases, I believe that his scenario is a consequence of fundamentally misunderstanding the function of DevOps, and of working at the wrong level of abstraction.

DevOps is about building user interfaces for developers.

If consultants are spending 90% of their time working out lower-level nuances of infrastructure orchestration then they have failed.

Just as the Ruby on Rails framework freed developers from having to make decisions about mundane implementation details that are universal to all web apps, various DevOps/enterprise PaaS solutions free you from having to architect from scratch basic and universal functionality like access management, log shipping, and exposing environment variables to applications.

The Rancher admin interface

These solutions include, but are not limited to:

• Cloud66
• RedHat OpenShift
• Cloud Foundry
• Rancher

Somewhere in the middle are also hosted platforms like Google App Engine, which give you a PaaS like experience (“app” abstractions, CLI, etc) but at better value and with fewer resource constraints than offerings like Heroku.

Cloud66 admin UI

Working with any such solution is a vastly different experience than working with low-level infrastructure automation tools like Chef, Ansible, the increasingly popular Terraform or any bespoke CLIs built around tools of this nature.

I find that an instructive litmus test of whether DevOps is delivering for a team is the ease with which a developer (not an infrastructure specialist) can “fork” (copy) an existing app and deploy to it. With high-level tools like OpenShift or Cloud66 this should be a relatively intuitive and quick task. With more bespoke tools it may not be something the developer is ever able to achieve by themselves.

Fundamentally, if your developers cannot work comfortably with your DevOps solution, the solution is a liability. I hope that this article has shed light on the critical distinction between infrastructure automation and higher-level PaaS, and that it saves organizations from going too far down the road of bespoke DevOps solutions where they may not be truly necessary or well-suited.

The post How to mess up DevOps: working at the wrong level of abstraction appeared first on Superset Blog.