Logging for Cloud Native Apps

2015-05-18 by Axel Fontaine

When it comes to building and running applications that are native to the cloud a number of old habits need to die and make way for modern alternatives. One of these is traditional file-based logging.

The days of using tail -f to watch logs scroll by in 5 different terminals are over.

This becomes even clearer as you move away from traditional operating systems and classic provisioning towards modern secure minimal images like those generated by CloudCaptain. These machines are tailor-made for running your application and don't contain any legacy cruft. In fact, they don't even let you log in as there is no ssh on board. All that is left, is the most efficient structure to run your application on VirtualBox and in the cloud.

So how do traditional instances compare to modern cloud native instances?

Traditional Instances	Cloud Native Instances
Long lived	Disposable
Reconfigured on change	Regenerated on change
Persistent storage	Ephemeral storage
Multiple GBs	Few MBs
Ability to log in via SSH	No SSH

Quite a different world indeed.

As you can see traditional file-based log rotation is not only outdated and risky (what if the instance's disk dies?), but also simply doesn't make sense anymore without the ability to log in.

However logging as such is still a very important tool. It is invaluable for system behavior and error root cause analysis. So how can it be done in this new cloud native world?

Logging needs

Here at CloudCaptain we view logs as great forensic tool in case of trouble. We do not see logging as a general purpose data sink. If you need structured access to large amounts of data in your logs, you should probably consider other data stores instead.

So what are the needs a logging solution must fulfill?

It must be durable for a certain duration (log value decreases exponentially with time)
It must be searchable (to quickly find related needless in the haystack)
It must be centralized (all instances of an application send their logs to the same place)
It must be integrated with the application (no mandatory clunky detours via disk or syslog)
It must be fast with little overhead (no substantial performance degradation)

Additionally we also want to be able to capture early system logs in case our application fails to start or can't connect to the centralized logging facility.

We therefore want to distinguish between instance boot logs (low volume, instance-specific, for diagnosing startup issues) and application logs (high volume, application-level, for diagnosing application behavior issues).

Instance Boot logs

The instance boot logs collect information from the earliest stages of instance startup (kernel boot) all the way to the point where the application is fully up and running. They are great for identifying environment-specific network and application configuration issues.

Luckily all virtualization and cloud platforms make this information available and if you are using CloudCaptain you have a unified and convenient way to access it. All you need to do is

> boxfuse logs instance-id

and you immediately see the boot log for that instance including DHCP lease acquiry and application startup details.

Application logs

On the application side of things there are number of excellent logging solutions available that fulfill all criteria described above. For JVM-based applications Loggly, Logentries and Papertrail make it a breeze as they all support direct Logback and Log4J integration.

Messages are sent directly by a log appender from memory via HTTPS to their backend connectors where they are ingested, aggregated, indexed and made available via nice web-based dashboards.

Let's look at one example in detail.

Integrating Logback with Loggly

Let's look at how simple it is to integrate two popular solutions: Logback (successor to Log4J) and Loggly (we could have just as easily picked Logentries or Papertrail).

Start by creating a free Loggly account and add a Java Logback source:

Now all you need to do is add the Loggly appender dependency to your build (it is available on Maven Central):

<dependency>
    <groupId>org.logback-extensions</groupId>
    <artifactId>logback-ext-loggly</artifactId>
    <version>0.1.2</version>
</dependency>

and configure the appender in your logback.xml (don't forget to make it async to avoid a performance hit):

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="loggly" class="ch.qos.logback.ext.loggly.LogglyAppender">
        <endpointUrl>https://logs-01.loggly.com/inputs/your-loggly-input-id</endpointUrl>
        <pattern>%d{ISO8601} %p %t %c{0}.%M - %m%n</pattern>
    </appender>
    <appender name="loggly-async" class="ch.qos.logback.classic.AsyncAppender">
        <appender-ref ref="loggly" />
    </appender>
    <root level="info">
        <appender-ref ref="loggly-async" />
    </root>
</configuration>

That's it! Almost immediately you will see the logs appearing in the Loggly UI:

You can now start search them, generate graphs and alerts, create custom dashboards and more!

Summary

Logging for cloud native applications is a breeze. Simply combine CloudCaptain for instance boot logs with modern hosted solutions like Loggly, Logentries or Papertrail and their Logback or Log4J integrations for your application logs and you will have a comprehensive, robust and highly available solutions up and running in no time.

Tired of snowflake servers? Create your CloudCaptain account now, start generating minimal and secure images in seconds and run them identically on VirtualBox and EC2.