CloudCaptain

Boxfuse is now CloudCaptain!

2022-01-17T00:00:00+00:00

The initial release of Boxfuse was in 2015. From then on Boxfuse’s feature set grew steadily as we added support for additional payload types, additional AWS regions, additional AWS instance families, additional app types and more!

Transition and Improvements

In 2019, we sold the Flyway part of our business and entered a long transition phase. During this time our attention was more divided than we would have liked, yet we still managed to deliver a number of major improvements including:

lightning fast AMI creation, usually in under 10 seconds
OpenJDK 11 and OpenJDK 17 support
m6i, r6i, c6i EC2 instances support
db.t4g RDS instances support

Introducing CloudCaptain

Today we are happy to announce that this transition phase has ended!

As part of this, please join us in saying goodbye to Boxfuse and welcoming CloudCaptain!

As you no doubt have noticed the old cloudcaptain.sh domain has now been phased out and replaced by our brand new cloudcaptain.sh where development will continue at an accelerated pace going forward!

We have lots of innovation in store and we can’t wait to share it with you!

Watch this space or follow @CloudCaptainSH on Twitter to stay updated!

Spring Boot 2 and t3 and r5 instances support

2018-08-23T00:00:00+00:00

Today CloudCaptain is introducing 2 important new features.

Spring Boot 2 support

Building upon CloudCaptain’s existing [Spring Boot 1.x support] (/blog/spring-boot-ec2), CloudCaptain now fully supports Spring Boot 2.0 and newer apps.

Spring Boot 2.0 is autodetected and CloudCaptain will then auto-configure ports, healthchecks and more based on the new property names introduced in Spring Boot’s 2.0 release.

T3 and R5 instance support

AWS recently released their new all hardware hypervisor called the Nitro System and along with it a number of new instance types making full use of it.

Today CloudCaptain expands its instance support to two new instances families, both running on the Nitro System.

First of all, CloudCaptain now supports the brand new t3 burstable instances. They are 10% more cost effective than t2 and run on more powerful hardware. All t2 users are encouraged to make the switch.

In addition to that, for RAM-heavy apps CloudCaptain now fully supports the new r5 instances. The are more cost effective, support more memory and come in a larger sizes at the top end as previous r4 and r3 instances. All r4 and r3 users are encouraged to make the switch.

Available today

Spring Boot 2 and t3 and r5 instance support is available today at no additional charge to all CloudCaptain customers. Enjoy!

And if you haven’t already, sign up for your CloudCaptain account now (simply log in with your GitHub id, it’s free), start deploying your application effortlessly to AWS today and have it running in minutes.

Introducing FastAMI

2018-03-28T00:00:00+00:00

When CloudCaptain deploys your apps to AWS, a two-step process takes places. First, CloudCaptain creates Amazon Machine Images, simply known as AMIs. Using these, EC2 instances can then be launched.

AMI creation has traditionally been rather time consuming. So much so that many AWS deployment technologies chose to forgo custom AMIs and all their benefits, and instead go for the more brittle process of launching an instance from a standard AMI and customizing it on startup.

Not so with CloudCaptain. Not only does CloudCaptain fully embrace immutable infrastructure and the benefits of no SSH, but today, building upon last month’s release of Private Vault we are unveiling a new fast and secure AMI provisioning mechanism which works across all supported AWS regions called CloudCaptain FastAMI.

Introducing CloudCaptain FastAMI

Today we are introducing CloudCaptain FastAMI: a transparent, fast and secure way to rapidly provision AMIs fully within your AWS account.

It works as follows: when CloudCaptain needs to create a new AMI it first launches a t2.micro instance in the default VPC of that region of your account. This instance then securily listens to AMI conversion requests. Once it receives such a request it pulls an image from your encrypted S3 private vault bucket and turns it into an encrypted AMI, all without the image data ever leaving your AWS account. This instance stays active until it hasn’t seen a new request for 60 minutes, at which point it terminates automatically.

This incurs a small cost of about 0.01 USD for AWS accounts that are no longer eligible for the AWS free tier. We felt this was a reasonable trade-off for all the the privacy, security and speed advantages of CloudCaptain FastAMI.

Available today

CloudCaptain FastAMI is available today at no additional charge to all CloudCaptain users who have already migrated their account to Private Vault. Support for this is fully transparent and initially available in the us-east-1and us-west-2 regions. The remaining AWS regions will be enabled in the coming weeks. Simply deploy images stored in your Private Vault to environments in any of those regions and you’ll automatically benefit from the privacy, security and speed advantages of CloudCaptain FastAMI. Enjoy!

So if you haven’t already, sign up for your CloudCaptain account now (simply log in with your GitHub id, it’s free), start deploying your application effortlessly to AWS today and have it running in minutes.

Introducing Private Vault

2018-02-12T00:00:00+00:00

CloudCaptain has always made it easy to to push and pull images to and from the CloudCaptain Vault. Up until now the CloudCaptain Vault was implemented as an S3 bucket within CloudCaptain’s account. This is a very convenient solution that makes it easy to get started. It does however come with a disadvantage that is increasingly important for our customers: even though their images are fully encrypted, they still lie within our AWS account and not theirs.

Introducing Private Vault

Today we are introducing the successor of the CloudCaptain Vault: the Private Vault. It retains the same ease of use as the CloudCaptain Vault, but it is now also comes with the added security and privacy advantage of being stored in an S3 bucket on your AWS account.

Private Vault is now enabled on all new CloudCaptain accounts by default.

Migrating from the CloudCaptain Vault to your Private Vault

Existing accounts can be migrated within a few seconds in the CloudCaptain Console. All you need to do is simply click the Upgrade to Private Vault button on the Overview tab. An encrypted S3 bucket will then be created in the default region for your AWS account (as selected under the globe on the overview tab in the CloudCaptain Console). All images you push from then on will automatically be stored in your new encrypted S3 bucket.

Existing images remain unaffected und can be accessed as before. They still live within the legacy CloudCaptain Vault and can be removed either manually via the Console or automatically on a least recently used basis when your Vault capacity has been exceeded.

This migration can take place at your best convenience until the 30th of June 2018. Starting July first, existing accounts that have not yet been migrated to Private Vault will automatically be migrated for you.

From then on any image stored in the legacy CloudCaptain Vault remains accessible as usual until the 31st of December 2018. After that the CloudCaptain Vault will be retired and any images it still contains will be deleted.

Available today

Private Vault is available today to all CloudCaptain users at no additional charge. Support for this is fully transparent. All you need to do is upgrade your account and upgrade your client to version 1.32.0.1441 or newer and you’ll automatically benefit from the privacy and security advantages of Private Vault. Enjoy!

Launch M5 and C5 instances on the Nitro System

2018-01-18T00:00:00+00:00

Six weeks ago at the last AWS re:Invent in Las Vegas, AWS introduced a new hypervisor that will pave the future of EC2. It moves the virtualization tasks traditionally performed by their previous hypervisor, Xen, and its management node, dom0 down into custom silicon and replaced the remaining bits with a very slim version of KVM.

This new hypervisor is called the Nitro System. It offers performance comparable to bare metal. And now that reserving capacity for a dom0 management node is no longer needed, it also exposes the entire capacity of the underlying hardware to EC2 instances. The first types of instances to run on the Nitro System and enjoy those benefits are the M5 and C5 EC2 instances.

This is a big change and it’s well worth familiraizing yourself with it. AWS did a great talk about it at re:Invent which is now also available online:

If you prefer to read, here is also a great article by Brendan Gregg from Netflix on the same subject: https://www.brendangregg.com/blog/2017-11-29/aws-ec2-virtualization-2017.html

Full CloudCaptain support

Starting today you can now use CloudCaptain to create AMIs and also launch M5 and C5 instances on the Nitro System in all regions where it is already deployed. To make this possible, CloudCaptain made a number of changes.

First of all we upgraded to the 4.14.14 Linux kernel. As part of this move we also open-sourced the kernel configuration used by CloudCaptain images and made in available on GitHub. Contributions and pull requests are welcome!

Next we added support for the two essential technologies required for AMIs running on the Nitro System: NVMe storage and Elastic Network Adapters. All new CloudCaptain images created with CloudCaptain client 1.31.0 and newer will now automatically include those drivers. The images are universal and able to run on Hyper-V, VirtualBox, EC2 with Xen and EC2 with Nitro, unchanged of course.

Finally we added support for the M5 and C5 instances, both to the CloudCaptain Client and the CloudCaptain Console.

Available today

This is available today at no additional charge. Support for this is fully transparent and in traditional CloudCaptain fashion, all the heavy lifting is done automatically for you behind the scenes. Our tests indicate that both instance types have the fastest launch and boot times of any EC2 instance type so far. Simply update your client and enjoy!

Introducing load-balanced-https apps

2017-10-04T00:00:00+00:00

CloudCaptain originally launched with two types of apps: single-instance for applications using Elastic IP addresses and load-balanced for applications using Classic Elastic Load Balancers. Over time we have added support for additional types of applications such as workers apps and one-off apps to support more diverse types of workloads.

Today CloudCaptain is expanding further with support for load-balanced-https apps using AWS Application Load Balancers, together with Auto Scaling Groups and optionally custom domains and TLS (SSL) certificates.

How does it work?

When you create a new load-balanced-https app using either the CloudCaptain Console

or the CloudCaptain Client

> boxfuse create my-app -app.type=load-balanced-https

CloudCaptain will be set up to automatically provision and configure all necessary resources when the app is deployed.

Just for the ALB this includes:

a new ALB (Application Load Balancer)
a security group with the correct permissions for the ALB
one or more listeners for the correct ports on the ALB
a target group associated with the ALB
rules connecting the listeners to the target group

Why Application Load Balancers?

AWS Application Load Balancers come with a number of important improvements for applications serving traffic over HTTP(S) including HTTP/2 and WebSockets, while building on the important features you already love like Auto Scaling and automatic TLS certificate management.

Seeing it in action

So let’s see how this works if we deploy our sample Spring Boot application on AWS:

> boxfuse run -app.type=load-balanced-https -env=test -tls.type=acm -domain=hello.boxfuse-example.com

Creating hello ...
Successfully created app hello (type: load-balanced-https, tls: acm, db: none, logs: cloudwatch-logs)
Fusing Image for hello.jar (Spring Boot) ...
Image fused in 00:03.673s (62850 K) -> myuser/hello:1.0
Pushing myuser/hello:1.0 ...
Verifying myuser/hello:1.0 ...
Waiting for AWS to create an encrypted AMI for myuser/hello:1.0 in eu-central-1 (this may take up to 50 seconds) ...
AMI created in 00:47.010s in eu-central-1 -> ami-75de631a
Creating security group boxsg-myuser-test-hello ...
Creating Log Stream boxfuse/test > myuser/hello ...
Creating Application Load Balancer for myuser/hello in test ...
Creating Target Group myuser-test-hello ...
Found certificate for hello.boxfuse-example.com => arn:aws:acm:eu-central-1:...
Creating ALB Listener for port 80 ...
Creating security group boxsg-myuser-test-hello-1.0 ...
Creating Launch Configuration boxlc-myuser-test-hello-1.0 ...
Creating Auto Scaling Group boxasg-myuser-test-hello-1.0 ...
Waiting for Auto Scaling Group boxasg-myuser-test-hello-1.0 to launch 1 t2.micro Instance ...
Auto Scaling Group: i-063c0eab44eb5d87c [Pending]
Auto Scaling Group: i-063c0eab44eb5d87c [InService]
Waiting for ALB to become active (this may take up to 10 minutes) ...
Waiting for ALB Target Group to put instances in service ...
ALB: i-063c0eab44eb5d87c [healthy]
Successfully running myuser/hello:1.0 in test at https://hello.boxfuse-example.com/

And our app is fully up and running, load balanced in an auto scaling group with a custom domain and an automatic TLS certificate.

All deploys from here on are performed with zero downtime by simply launching a new auto scaling group and switch out the old one within the target group.

BYOTG (Bring Your Own Target Group)

Advanced users also have the option to configure their own ALB and Target Group. From there all that is needed is simply pointing CloudCaptain at the name of the desired Target Group and CloudCaptain will use that one instead of the ALB and Target Group auto provisioning described above.

Available today

Support for load-balanced-https apps is available today at no additional charge on all paid CloudCaptain plans. Enjoy!

Customer Interview: KwiqJobs

2017-09-11T00:00:00+00:00

Today we are kicking off a series of interview where CloudCaptain customers talk about their experiences. In this first interview I chatted with Jannes Stubbemann, the CTO of KwiqJobs.

Hi Jannes, can you tell us about yourself and your company?

KwiqJobs wants to turn the waiting time of people into a new resource. We are building a mobile marketplace for microjobs that allows companies to gather data from our app users by placing jobs on our platform. These jobs vary from market research questionnaires to tasks in which users label or generate data that later can be used to train an artificial intelligence. For doing these jobs, which take a few seconds only, the users earn a few amount of money. My position is the one of the CTO. I am a proud co-founder of KwiqJobs and joined the team on a startup weekend in June last year. Currently we are about to close our seed investment and are building our developer team in a remote setup.

That sounds great! What does your infrastructure setup look like?

Our product as such consists of two main pieces, the mobile application (for now only Android) and a Scala/Play Application as the backend. The backend is currently hosted in AWS where it gets deployed by CloudCaptain. As our version control system we are using git in combination with GitLab to organize our work and our continuous delivery pipelines. As we are working in a remote setup, we use slack as our main communication tool. It helps us not only to communicate with each other but also to get notified about app crashes via crashlytics or deployments from GitLab.

Why did you choose CloudCaptain?

I think one of the key success factors of good software development is a good and easy to manage continuous delivery pipeline. As a startup we don’t have the capabilities to set it all up using e.g. Docker. Instead, we where looking for a managed solution that allowed us to have a working cd pipeline from the very beginning. When we where looking for such a solution we found CloudCaptain. It seemed really well suited and so far it hasn’t disappointed us. There is no need for us to care about the AWS environment as such and we can focus on the development of our platform instead.

Can you tell us a bit more on how you are using CloudCaptain and what your favorite feature is?

We use CloudCaptain for our continuous delivery of the backend. Whenever code is merged into our develop or master branch, GitLab ci is running CloudCaptain in order to deploy the application in our test or production environment on AWS. My favorite feature is that with CloudCaptain we have one central program, even more one single command, to manage multiple things around our application. For instance we didn’t need to integrate New Relic by hand, CloudCaptain has first level support for that already! We just needed to include it in the command. Or the logs which are also set up by CloudCaptain and can even be accessed via it. This helps us to focus on the development of our application. We don’t get distracted by caring how to setup New Relic or accessing our EC2 instances to change some settings. It’s so many things in one place and easy to use which make it a really powerful tool particular for startups.

After we built our mvp and onboarded our first customer in the last year, it is our goal this year to build a scalable application and onboard more customers that we have already in our pipeline.

Thanks!

This concludes our first interview. Stay tuned for more!

One-off apps

2017-09-04T00:00:00+00:00

When it comes to running production apps, CloudCaptain offers a range of options, from single-instance to load-balanced and worker apps. They cover a wide range of scenarios from simple hobby apps to complex and robust robust auto-scaling auto-recovering production workloads.

But sometimes all you need is a single run of an administrative or maintenance task.

Introducing one-off apps

Today we are introducing a new type of app: one-off

This is an app that starts, does its thing and once it is done, the application process can safely exit and the instance will terminate. This is particularly useful for administrative or maintenance workloads where you just want a single task to complete and that’s it.

Typical uses would be one-time data exports/imports, data transformation, batch operations or ad-hoc system maintenance.

How does it work?

Say you have a maintenance job packaged as a jar file. You can simply run it on your prod environment as follows:

> boxfuse run my-maintenance-job-1.0.jar -app.type=one-off -env=prod

This will launch an AWS instance in your prod with environment with your job running. Once the main method terminates or the application exits with System.exit() the instance will automatically shut down and AWS will stop charging you for it.

Available today

Support for one-off apps is available today at no additional charge on all paid CloudCaptain plans. Enjoy!

Hello Hyper-V!

2017-06-28T00:00:00+00:00

CloudCaptain lets you test your images rapidly and with great fidelity by allowing you to run them unchanged both locally and on AWS. To run your images locally CloudCaptain supports VirtualBox. It’s easy to install and it’s available for Windows, macOS and Linux.

Windows 10 Pro however already ships with its own built-in hypervisor: Hyper-V. This is a great feature. It is however only available on the Pro edition of Windows 10 (Home edition is not supported) Not only that, but once it is active no other hypervisors are allowed to run. This effectively means you need to disable Hyper-V (and reboot your machine) every time you want to use VirtualBox.

No more! Starting today, CloudCaptain now also comes with first-class Hyper-V support on Windows 10 Pro.

How it works

When running on Windows, by default CloudCaptain will autodetect whether you are using Windows 10 Pro and whether Hyper-V is active. If that’s the case, CloudCaptain will automatically select Hyper-V for running instances in the local dev environment.

Otherwise it will simply use VirtualBox as it always has.

You can also explicitly influence this behavior by setting the new platform configuration property to either hyperv to force Hyper-V usage or virtualbox to force VirtualBox usage.

Available today

The CloudCaptain support for Hyper-V is available today for all Windows 10 Pro users. Enjoy!

Add Custom Tags to your AWS Resources

2017-06-09T00:00:00+00:00

AWS lets you associate useful metadata with your provisioned resources in the form of tags. By default CloudCaptain uses this to consistently tag your AMIs, EBS snapshots, EBS volumes, ELBs, Instances, Auto Scaling Groups, Launch Configurations, RDS databases, RDS snapshots and more.

CloudCaptain does so using a combination of the following tags:

AWS Tag Name	Description	Example
`boxfuse:env`	The CloudCaptain environment	prod
`boxfuse:app`	The CloudCaptain application	myuser/myapp
`boxfuse:image`	The CloudCaptain image	myuser/myapp:1.2.3
`Name`	(For EC2 instances only) The CloudCaptain image and environment	boxfuse myuser/myapp:1.2.3@prod

Introducing Dead-Easy Custom Tags

In addition to the standard tags automatically created by CloudCaptain you can now also specify one or more custom tags that should be applied to all created AWS resources that CloudCaptain creates.

Typical Uses

Being able to add custom tags is particularly useful in a number of scenarios:

Cost allocation in your monthly AWS bill
Creating Resource Groups in the AWS Console
Automatic use of spot instances with AutoSpotting

And much more!

How it works

For each app in each environment you can now define a set of custom tags that should be added in addition to the existing CloudCaptain ones. To achieve this you can either use the CloudCaptain Console:

Or if you prefer you can also simply use the CloudCaptain client:

> boxfuse cfg myapp -env=prod -tags.team=frontend-dev -tags.spot-enabled=true

And that’s it! All new AWS resources CloudCaptain now provisions in the prod environment for myapp will now receive these two custom tags in additional to the standard CloudCaptain ones.

Available today

The CloudCaptain support for custom tags is available today at no additional charge on all paid plans. Enjoy!

Automatic TLS (SSL) Certificates

2017-04-26T00:00:00+00:00

This month CloudCaptain turned 2 and it's come a long way since its humble beginnings! In fact only since last year's anniversary we have added a lot of major new features including custom JREs, kernel tuning, live reloading, worker apps, Go and Revel support, New Relic integration, CloudWatch Logs integration, generic Linux x64 apps, custom domains and custom Elastic IPs. And that's of course on top of the countless minor features and polish which have been integrated in the 62 CloudCaptain Client updates released in the past year alone!

Today we are kicking of CloudCaptain's third year with a very highly requested new feature: automatic TLS (SSL) certificate management and renewal.

How does it work?

Load balanced apps using a custom domains can now be configured to automatically expose themselves via HTTPS with a TLS (SSL) certificate issued by the AWS Certificate Manager.

CloudCaptain configures the ELB to terminate the client-facing TLS connection with a valid TLS certificate and forward the request to your backend instances using TLS connection with a self-signed certificate.

Once set up, the certificates auto-renew before they expire without any action required on your behalf. This ensures clients always see a green lock in the browser while at the same time always ensuring all data in motion travels encrypted.

Seeing it in action

Let's see what this looks like in action. For this post, we're going to deploy a Spring Boot application with HTTPS at boot.boxfuse-example.com, which will be a new subdomain of the boxfuse-example.com domain which we own.

Obtaining our certificate

First, let's get the certificate part out of the way. To do so we're going to the AWS Certificate Manager page of the AWS Console and we request a wildcard certificate for our domain:

Once we have received the email from AWS requesting us to confirm our ownership of the domain for the certificate, we click on the approval link:

Our wildcard certificate is now ready to use.

Creating the app

We now need a basic Spring Boot app to deploy. For this we're going to generate one with the Spring Initializr and add a basic controller. To do this, we simply follow the steps described in our Spring Boot getting started guide.

Once that is done, we go ahead and build it:

> mvnw package

Running our application

All we need to do now is run our application on AWS using our new subdomain (which will be auto-created) and our freshly created wildcard certificate. To do so we make sure the app.type, tls.type and domain parameters are set correctly:

> boxfuse run -app.type=load-balanced -tls.type=acm -domain=boot.boxfuse-example.com -env=prod

Creating getstarted-springboot ...
Successfully created app getstarted-springboot (type: load-balanced, tls: acm, db: none, logs: cloudwatch-logs)
Fusing Image for getstarted-springboot-1.0.jar (Spring Boot) ...
Image fused in 00:03.320s (59689 K) -> axelfontaine/getstarted-springboot:1.0
Pushing axelfontaine/getstarted-springboot:1.0 ...
Verifying axelfontaine/getstarted-springboot:1.0 ...
Waiting for AWS to create an encrypted AMI for axelfontaine/getstarted-springboot:1.0 in eu-central-1 (this may take up to 50 seconds) ...
AMI created in 00:28.047s in eu-central-1 -> ami-161cc079
Creating security group boxsg-axelfontaine-prod-getstarted-springboot ...
Creating Log Stream boxfuse/prod > axelfontaine/getstarted-springboot ...
Found certificate for boot.boxfuse-example.com => arn:aws:acm:eu-central-1:103037756454:certificate/5ba59f97-5447-4b91-bb81-fa5729d80331
Creating Elastic Load Balancer for getstarted-springboot in prod ...
Creating security group boxsg-axelfontaine-prod-getstarted-springboot-1.0 ...
Creating Launch Configuration boxlc-axelfontaine-prod-getstarted-springboot-1.0 ...
Creating Auto Scaling Group boxasg-axelfontaine-prod-getstarted-springboot-1.0 ...
Waiting for Auto Scaling Group boxasg-axelfontaine-prod-getstarted-springboot-1.0 to launch 1 t2.micro Instance ...
Auto Scaling Group: i-050ff206b0370cefe [Pending]
Auto Scaling Group: i-050ff206b0370cefe [InService]
Waiting for ELB to put instances in service ...
ELB: i-050ff206b0370cefe [OutOfService] => Instance is in pending state.
ELB: i-050ff206b0370cefe [OutOfService] => Instance registration is still in progress.
ELB: i-050ff206b0370cefe [InService]
Successfully running axelfontaine/getstarted-springboot:1.0 in prod at https://boot.boxfuse-example.com/

And here is our app up and running with a valid TLS (SSL) certificate and a green lock in the browser:

And that's it! CloudCaptain automatically created and configured the AMI, the security groups, the Elastic Load Balancer, the CloudWatch Logs stream, the Auto-Scaling group and the instances and everything is fully set up and secure. And best of all, our TLS (SSL) certificate will be automatically renewed and updated before it expires without any action required on our behalf.

Available today

The CloudCaptain support for automatic TLS (SSL) certificate management and renewal for load-balanced apps with custom domains is available today at no additional charge on all paid plans. Enjoy!

So if you haven't already, sign up for your CloudCaptain account now (simply log in with your GitHub id, it's free), start deploying your application effortlessly to AWS today and have it running online with SSL in minutes.

Custom Elastic IPs

2017-03-02T00:00:00+00:00

When you need your application to have a unique public IP address that doesn't change over time, CloudCaptain lets you create your application as single-instance to ensure it will always be deployed behind the same AWS Elastic IP at runtime.

Until now, CloudCaptain has always automatically provisioned that Elastic IP address (along with all other AWS resources) for you. This is extremely convenient and by far the best choice for the vast majority of the single-instance applications.

There are however a number of cases where this falls short. The most common one is being able to know the exact IP address in advance to ensure the necessary firewall or security group rules have been set up correctly. This is especially relevant when an application doesn't need to be up 24/7. So far killing an application meant terminating all related AWS resources, including the Elastic IP. When running it again the application would get the same boxfuse.io or custom domain, however a new Elastic IP would be created and the DNS A record would then point at that one instead.

Today we are introducing support for custom Elastic IPs for single-instance apps to fix this.

How does it work?

First of all, ensure you have allocated the Elastic IP you need either via the AWS Console or some other client.

From here on things couldn't be easier. Simply go to your application in the CloudCaptain Console and configure it for the environment you want:

And now simply select the Elastic IP of your choice from the list and your app will be deployed behind it.

Needless to say, once deployed the boxfuse.io or custom domain will also point directly at that Elastic IP.

Using the CloudCaptain Client

If you prefer to use the CloudCaptain Client instead, you can simply specify the Elastic IP as follows:

> boxfuse run -env=prod -elasticip=35.157.139.99

And for infrastructure as code simply place it in your boxfuse.conf file which you can check in to version control along with your sources:

elasticip=35.157.139.99

Available today

The CloudCaptain support for custom Elastic IPs for single-instance apps is available today at no additional charge on all paid plans. Enjoy!

Custom Domains

2017-02-24T00:00:00+00:00

To get your JVM, Node.js or Go application up and running quickly and effortlessly on AWS all you need to do is type

> boxfuse run -env=prod

This will provision all necessary resources, deploy your application and make it available under a brand new boxfuse.io subdomain like myapp-mygreatcompany.boxfuse.io. But wouldn't it be great if you could make it available under myapp.com or myapp.mygreatcompany.com instead?

Until now this meant manually creating a new CNAME record in your DNS that would point to your boxfuse.io subdomain which in turn would point to the underlying AWS resource.

No more. Today we are introducing dead easy first-class support for custom domains using AWS Route 53.

How does it work?

First of all, ensure you have registered your domain using Route 53 or that you are using Route 53 for managing your domain's DNS.

From here on things couldn't be easier. Simply go to your application in the CloudCaptain Console and configure it for the environment you want:

And now simply select the domain of your choice from the list and your app will be deployed at mygreatcompany.com. And if you want to use a new subdomain that's easy too. Simply type the name of the subdomain in the textbox and voila! Your app will be deployed at myapp.mygreatcompany.com.

Those domains now point directly at the underlying AWS Elastic IP or Elastic Load Balancer.

Using the CloudCaptain Client

If you prefer to use the CloudCaptain Client instead, you can simply specify the domain as follows:

> boxfuse run -env=prod -domain=myapp.mygreatcompany.com

And if you lean towards infrastructure as code simply place it in your boxfuse.conf file which you can check in to version control along with your sources:

domain=myapp.mygreatcompany.com

Available today

The CloudCaptain built-in support for custom domains using AWS Route 53 is available today at no additional charge on all paid plans. Enjoy!

Shipping native Linux x64 binaries and libs with JVM apps

2017-01-24T00:00:00+00:00

While CloudCaptain makes it dead easy to run your JVM applications (Spring Boot, JHipster, Grails, Dropwizard, Tomcat, TomEE or executable jar) on AWS, sometimes those applications also depend on other native Linux x64 binaries and libs. Typically these would be native applications for audio, video or image processing, but they could be anything.

Until now this meant writing some custom code to manually extract those binaries to the filesystem, giving them execute permissions and so on. Not exactly complicated. But certainly more complicated than things could and should be.

No more. Today we are introducing dead easy built-in support for shipping native Linux x64 binaries with your JVM apps.

How does it work?

Simply place your binaries under a special native/bin directory and CloudCaptain will automatically add them to the PATH at runtime in your instances.

If those binaries also depend on additional shared libraries beyond the C library, place the .so files of your libraries under native/lib and CloudCaptain will automatically add them to the LD_LIBRARY_PATH at runtime in your instances.

And that's all!

Seeing in action

For this post, we're going to build a remote version of the Linux cowsay utility using Spring Boot.

First let's start by taking the sources of this simple c port of cowsay:

#include <stdio.h>

int main(int argc, char **argv)
{
  int i;

  if (argc == 1)
    printf("< moOh >\n");
  for (i = 1; i < argc; i++)
    if (i == 1)
      printf("/ %s \\\n", argv[i]);
    else if (i == argc - 1)
      printf("\\ %s /\n", argv[i]);
    else
      printf("| %s |\n", argv[i]);
  printf("  \\ ^__^\n");
  printf("    (oo)\\_______\n");
  printf("    (__)\\       )\\/\\\n");
  printf("        ||----w |\n");
  printf("        ||     ||\n");
  return (0);
}

and compiling them into a native Linux x64 binary:

$ gcc -Wall -g cowsay.c -o cowsay.elf64

Now let's create a simple Spring Boot app to expose this:

$ curl 'https://start.spring.io/starter.zip?type=maven-project&bootVersion=1.4.3.RELEASE&baseDir=remote-cowsay&groupId=com.boxfuse.demo&artifactId=remote-cowsay&version=1.0&name=remote-cowsay&description=Remote+cowsay&packageName=com.boxfuse.demo&packaging=jar&javaVersion=1.8&language=java&autocomplete=&generate-project=&style=web' -o remote-cowsay.zip && unzip remote-cowsay.zip

Next add our freshly compiled cowsay.elf64 Linux x64 binary to our project under the src/main/resources/native/bin directory to ensure it will be added to the PATH at runtime:

 remote-cowsay
   src
     main
       java
         com
           boxfuse
             demo
               RemoteCowsayApplication.java
         native
   bin
     cowsay.elf64
     test
   .mvn
   .gitignore
   mvnw
   mvnw.cmd
   pom.xml

And finally let's add a controller to our RemoteCowsayApplication class:

package com.boxfuse.demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.util.FileCopyUtils;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

import java.io.IOException;
import java.io.InputStreamReader;
import java.nio.charset.StandardCharsets;

@SpringBootApplication
@RestController
public class RemoteCowsayApplication {
    @RequestMapping(path = "/")
    public String cowsay(@RequestParam(value = "t", defaultValue = "Moo") String text) throws IOException {
        return FileCopyUtils.copyToString(new InputStreamReader(
            // No need to specify absolute path as binary is available on $PATH
            new ProcessBuilder("cowsay.elf64", text).start().getInputStream(),
            StandardCharsets.UTF_8));
    }

    public static void main(String[] args) {
        SpringApplication.run(RemoteCowsayApplication.class, args);
    }
}

Now it is time to package our jar:

$ mvnw package

And let CloudCaptain create an image and launch an instance on VirtualBox:

$ boxfuse run

Creating remote-cowsay ...
Mapping remotecowsay-dev-axelfontaine.boxfuse.io to 127.0.0.1 ...
Successfully created app remote-cowsay (type: single-instance, db: none, logs: cloudwatch-logs)
Fusing Image for remote-cowsay-1.0.jar (Spring Boot) ...
Image fused in 00:03.320s (58621 K) -> axelfontaine/remote-cowsay:1.0
Launching Instance of axelfontaine/remote-cowsay:1.0 on VirtualBox ...
Forwarding http port localhost:8080 -> vb-f3f3a0a4:8080
Instance launched in 00:04.597s -> vb-f3f3a0a4
Waiting for payload to start on vb-f3f3a0a4:8080 (expecting HTTP 200 at / within 300s) ...
Successfully started payload in 00:12.134s -> https://127.0.0.1:8080

Last but not least, let's see it in action!

$ curl https://localhost:8080?t=Moooooohh
/ Moooooohh \
  \ ^__^
    (oo)\_______
    (__)\       )\/\
        ||----w |
        ||     ||

And there you have it! Our native Linux x64 binary was automatically added to the PATH with the correct execute permissions. And all we needed to do to invoke it was a simple one-liner.

Available today

The CloudCaptain built-in support for shipping native Linux x64 binaries with your JVM apps is available today at no charge to all customers. Enjoy!

Logback and Log4J2 appender for AWS CloudWatch Logs

2016-12-09T00:00:00+00:00

Logging is one of the major diagnostic tools we have at our disposal for identifying issues with our applications. Traditionally logging consistent of writing lines of text to a file on the local filesystem. This creates a bunch of issues. Not only does one now have to start hunting through various backend servers to see which one actually processed the request we were looking for. Without forgetting of course that multiple requests from the same session may land on different backend instances. But much more severe is the fact that in a cloud world any persistent data on an instance dies together with the instance. And in the case of auto-scaling we don't even actively control anymore when certain instances are terminated.

To remedy this we need to look at a new solution: centralized logging. All logs get sent to a central service where they are aggregated, stored, and made searchable. AWS offers its own service for this called CloudWatch Logs.

We first added support for it in October, by letting you create applications and indicate that you want your logs sent to CloudWatch Logs. Images of those applications automatically redirect both stdout and stderr using the open-source CloudCaptain CloudWatch Logs agent to the CloudWatch Logs LogStream of the application within the CloudWatch Logs LogGroup for the current environment. So if you haven't already, pause here and read that article first.

Today we are taking things further for JVM apps by offering you a brand-new open-source appender for both Logback and Log4J. This gives you much more fined-grained control by integrating natively into your preferred logging framework and letting you configure exactly how you want to route your logs.

Installation

To include the CloudCaptain Java log appender for AWS CloudWatch Logs in your application all you need to do is include the dependency in your build file.

Maven

Start by adding the CloudCaptain Maven repository to your list of repositories in your pom.xml:

<repositories>
    <repository>
        <id>central</id>
        <url>https://repo1.maven.org/maven2/</url>
    </repository>
    <repository>
        <id>boxfuse-repo</id>
        <url>https://files.cloudcaptain.sh</url>
    </repository>
</repositories>

Then add the dependency:

<dependency>
    <groupId>com.boxfuse.cloudwatchlogs</groupId>
    <artifactId>cloudwatchlogs-java-appender</artifactId>
    <version>1.0.2.17</version>
</dependency>

Gradle

Start by adding the CloudCaptain Maven repository to your list of repositories in your build.gradle:

repositories {
    mavenCentral()
    maven {
        url "https://files.cloudcaptain.sh"
    }
}

Then add the dependency:

dependencies {
    compile 'com.boxfuse.cloudwatchlogs:cloudwatchlogs-java-appender:1.0.2.17'
}

Usage

To use the appender you must add it to the configuration of your logging system.

Logback

Add the appender to your logback.xml file at the root of your classpath. In a Maven or Gradle project you can find it under src/main/resources:

<configuration>
    <appender name="CloudCaptain-CloudwatchLogs" class="com.boxfuse.cloudwatchlogs.logback.CloudwatchLogsLogbackAppender"/>

    <root level="debug">
        <appender-ref ref="CloudCaptain-CloudwatchLogs" />
    </root>
</configuration>

Log4J2

Add the appender to your log4j2.xml file at the root of your classpath. In a Maven or Gradle project you can find it under src/main/resources:

<?xml version="1.0" encoding="UTF-8"?>
    <Configuration packages="com.boxfuse.cloudwatchlogs.log4j2">
    <Appenders>
        <CloudCaptain-CloudwatchLogs/>
    </Appenders>
    <Loggers>
        <Root level="debug">
            <AppenderRef ref="CloudCaptain-CloudwatchLogs"/>
        </Root>
    </Loggers>
</Configuration>

Code

And that's all the setup you need! You can now start using it from code as you normally would using either SLF4J or the api of your choice.

Logging a message like this:

private static final Logger LOGGER = LoggerFactory.getLogger(MyClazz.class);
...
LOGGER.info("My log message ...");

will now automatically sent it to CloudWatch Logs in structured form as a JSON document. The CloudCaptain CloudWatch Logs appender automatically adds critical metadata to the document as well so that it looks like this:

{
    "image": "myuser/myapp:123",
    "instance": "i-607b5ddc",
    "level": "INFO",
    "logger": "org.mycompany.myapp.MyClazz",
    "message": "My log message ...",
    "thread": "main"
}

This is very useful as this will allow us to query and filter the logs later.

Note that these are just the attributes sent automatically. You also have the possibility to fill in many other ones (such as current user, session id, request id, ...) via the MDC as described in the docs of the GitHub project.

Displaying the Logs

To display the logs simply open a new terminal and show the logs for your app in your desired environment:

> boxfuse logs myapp -env=prod

Live tailing

And if you want to follow along in real time you can use log tailing:

> boxfuse logs myapp -env=prod -logs.tail

And new logs will now automatically be displayed as soon as they are sent from the application to CloudWatch Logs.

Log filtering

This however can produce a lot of output. To aid finding the needle in the haystack, CloudCaptain also supports powerful log filtering, both on existing logs as well as during live tailing of a log stream.

You can make use of the attributes of the structured logs to filter them exactly the way you like. For example, to tail the logs live on the prod environment on AWS and only show the logs for a specific instance all you need to do is:

> boxfuse logs myapp -env=prod -logs.tail -logs.filter.instance=i-607b5ddc

And if you aren't quite sure what you are looking for you can also simply filter by time. For example to show all the logs created in the last minute (60 seconds) you could do:

> boxfuse logs myapp -env=prod -logs.tail -logs.filter.start=-60

This is a very powerful feature and there are many more filters which can all be combined. You can find all the details in the documentation.

Available today

The CloudCaptain Logback and Log4J2 Appender for CloudWatch Logs is available today. It is fully open-source (Apache 2.0 license) and you can browse the sources as well as read the detailed documentation on the GitHub project.

This feature is available at no charge to all customers.

Deploy any Linux x64 app on AWS with one command

2016-11-16T00:00:00+00:00

CloudCaptain is all about effortlessly deploying your applications so they just work. This is made possible by a combination of intelligent minimal image creating using deep integration with popular frameworks and language runtimes as well as seamless orchestration of blue/green deployments on AWS.

There are however a number of languages and frameworks we don't specifically support yet. So far this meant that if you weren't using the JVM, Node.js or Go you were out of luck.

Today we are introducing generic Linux x64 apps support. You can now deploy any Linux x64 app you like and enjoy CloudCaptain's great AWS integration including ultra-fast AMI creation, blue/green deployment, auto-scaling and more.

Seeing it in action

Let's dive in and look at how you can deploy any Linux x64 app using CloudCaptain.

To demonstrate this we are going to run Nginx on AWS. To do so we'll need two things:

a tar.gz file containing the app we want to run (in this case Nginx)
the command required to start the app

For this demo we'll use a Linux x64 build of Nginx available from this page on GitHub.

To fuse an image and run it on AWS all we need to do is pass the command to start it and which port to open:

> boxfuse run nginx-static-1.5.tgz "-cmd=nginx-static/sbin/nginx -g 'daemon off; user root;'" -ports.http=80

Creating nginx-static ...
Mapping nginxstatic-dev-myuser.boxfuse.io to 127.0.0.1 ...
Created app nginx-static (type: single-instance, db: none, logs: none)
Fusing Image for nginx-static-1.5.tgz (Linux x64) ...
Image fused in 00:00.707s (10522 K) -> myuser/nginx-static:1.5
Pushing myuser/nginx-static:1.5 ...
Verifying myuser/nginx-static:1.5 ...
Waiting for AWS to create an AMI for myuser/nginx-static:1.5 in eu-central-1 (this may take up to 50 seconds) ...
AMI created in 00:26.849s in eu-central-1 -> ami-d2dc19bd
Creating security group boxsg-myuser-prod-nginx-static ...
Creating Elastic IP ...
Mapping nginxstatic-myuser.boxfuse.io to 35.156.101.145 ...
Creating security group boxsg-myuser-prod-nginx-static-1.5 ...
Launching t2.micro instance of myuser/nginx-static:1.5 (ami-d2dc19bd) in prod (eu-central-1) ...
Instance launched in 00:24.478s -> i-8f39a332
Creating Cloud Watch Alarm for Instance auto-recovery -> i-8f39a332-auto-recovery-alarm
Waiting for AWS to boot Instance i-8f39a332 and Payload to start at https://54.93.62.223/ ...
Payload started in 00:00.003s -> https://54.93.62.223/
Associating Elastic IP 35.156.101.145 to i-8f39a332 ...
Waiting 15s for AWS to complete Elastic IP Zero Downtime transition ...
Successfully deployed myuser/nginx-static:1.5 in prod at https://nginxstatic-myuser.boxfuse.io/

And here we go! We created a 10 MB image containing a boot loader, the Linux kernel and Nginx in 0.7 seconds. CloudCaptain then deployed it on AWS by creating all necessary infrastructure including a custom domain name, an elastic IP, security groups, an AMI, a t2.micro instance and more in under 60 seconds total.

And here it is up and running:

All of this in under one minute from start to finish with just a single command. And best of all, all subsequent updates will always be performed using zero downtime blue/green deployments.

Summary

CloudCaptain now let's you deploy any Linux x64 app you want with the same orchestration features already available to JVM, Node.js and Go apps including ultra-fast AMI creation, blue/green deployment, auto-scaling and more.

Simply package your app as a tar.gz, pass the command required to start it and CloudCaptain will get you up and running on AWS in no time.

Check out the documentation and if you haven't already, sign up for your CloudCaptain account now (simply log in with your GitHub id, it's free) and start deploying any Linux x64 application effortlessly to AWS today.

Monitoring with New Relic

2016-10-24T00:00:00+00:00

Application and server monitoring provide invaluable insights into the health and availability of your apps. Today we are introducing first-class support for New Relic with automatic installation and configuration of both the New Relic Servers Linux x64 agent and the New Relic Java agent.

Server Monitoring

New Relic's free server monitoring provides you with in-depth monitoring of CPU usage, load average, physical memory, disk I/O, network I/O and processes.

Installation

To enable this for your app, simply pass in your New Relic license key when fusing an image:

> boxfuse fuse -newrelic.licensekey=0123456789abcdef0123456789abcdef01234567

CloudCaptain will then automatically install the New Relic Servers Linux x64 agent and configure it to start reporting to your New Relic account.

Application Monitoring (JVM apps only)

To provide you even deeper insights, New Relic also comes with monitoring at the application level with detailed analysis of transactions, throughput, database queries and much more. Additionally you can also define alerts to get notified of performance degradations and availability issues with your app.

Installation

For JVM apps, when supplying your New Relic license key as described above, CloudCaptain will also automatically install and configure the New Relic Java agent.

The application will then begin reporting to New Relic as myapp (myenv). In case of an application named hello deployed in the prod environment this would then be hello (prod).

Alternatively you can also supply a newrelic.yml configuration file for the Java agent and CloudCaptain will automatically use that instead. CloudCaptain will then install the agent for you, but won't override any application name you may have configured. If you haven't configured a New Relic license key as described above, CloudCaptain will use the license key contained in your newrelic.yml configuration file instead.

Summary

CloudCaptain now has first-class support for New Relic. This provides you with a robust and in-depth monitoring service for both your instances and the apps running on them.

All you need to do is literally pass your New Relic license key when fusing an image and the necessary New Relic Servers Linux x64 and New Relic Java agents will be automatically installed and configured for you.

This new feature is available at no charge to all CloudCaptain users.

So if you haven't already, sign up for your CloudCaptain account now (simply log in with your GitHub id, it's free) and start deploying your applications effortlessly to AWS in minutes and enjoy the great in-depth monitoring with New Relic.

Introducing AWS CloudWatch Logs support with Live Tailing

2016-10-07T00:00:00+00:00

Logging provides critically important visibility into the inner workings of our applications and offers a great way to perform both live and post-mortem analysis of interactions with our systems.

To be able to enjoy those benefits effectively a logging solution must possess a number of important qualities including durability, security, availability, searchability and speed.

We must also distinguish between two important types of logs: instance boot logs and application logs.

Instance boot logs collect information from the earliest stages of instance startup (bootloader and kernel boot) all the way to the point where the application is fully configured and ready to start shipping log events to the logging service of our choice. This makes instance boot logs great for identifying early environment-specific network and application configuration issues that may prevent an application from coming up correctly and start logging them in the first place.

Application logs are all the logs produced by our application from that point on.

For the latter CloudCaptain provided no out-of-the-box solution until now and you had to rely on external solutions to fill that gap. Today we are introducing first class AWS CloudWatch Logs integration to close this gap.

So without further ado, let's dive in and look at it in more detail!

Centralized Logging

In a modern microservices and cloud architecture, a single client request can end up traversing multiple services. Each of those services is then auto-scaled across a number of machines to have the appropriate capacity to handle the current load at all times.

This has a number of serious implications:

If more than one service is triggered by the request, more than one machine will produce logs related to that request.
The exact machine which will handle that specific request for each service is determined in real time by the load-balancer and not known in advance.
A machine which may have handled requests a few minutes ago, may now be terminated due to a change in load of the system having triggered a scale-in activity.

Additionally in the world of immutable infrastructure and minimal images there is a strong focus on security and the principle of least privilege, which also means there is no SSH on board of the running instances.

These realities effectively mean that application logs cannot be stored on the instance itself. Instead they must be shipped to a centralized logging service where they can then be securely stored and made available.

This is a great solution as it allows us to centrally enforce, enable and control a number of things.

Durability

Logs must be retained for the time required to perform useful analysis on them. Their value often decreases sharply over time. This stands in stark contrast to the space they occupy and the cost associated with it. It is therefore crucial to choose an appropriate retention period to strike the right balance between storage cost and historical usefulness.

The other important aspect of having a centralized logging service is enabling logs to outlive the machines that produced them.

Security

It is often forgotten that logs usually contain a lot of sensitive information. It crucial that the protection of this information be held to the same standards as the information present in our other systems. This means there is a shared responsibility here. On the one hand it is the responsibility of the centralized logging service to provide strong access controls. On the other hand, it is also your responsibility to carefully evaluate whether certain kinds of information maybe shouldn't be obfuscated or left out of the logs in the first place.

Availability

Since a load-balancer may send a request to any instance of a service and that service may in turn call many other services, a centralized logging services is a life-saver for actually finding the logs related to that request. No more searching for a needle in a haystack. Instead all the logs are available from one central place. Equally important here of course is the fact that due to auto-scaling events some of the machines which have produced logs may be long gone by the time someone is ready to analyse them.

Searchability

A centralized logging service makes it trivial to search all logs in one go. This includes all the logs produced by all instances of all services within an environment. This makes it possible to easily obtain all the logs for things like:

a single request across services
all sessions for a single user across one or more services
all instances of an application

And many more!

Speed

Of course to be usable a centralized logging service must also come with logging agents installed on the clients that produce as little runtime performance overhead as possible. For this it is essential that the sending of logs events to the service be asynchronous and non-blocking to the regular request processing flow. In a lot of cases today the network is actually very reliable and fast. Smart batching in memory virtually eliminates the need for on-disk buffering which is often slower than sending the events directly over network as the disks often aren't local anyway!

AWS CloudWatch Logs

So far in order to use a centralized logging service with CloudCaptain one had to rely on an external service like Loggly, Logentries or Papertrail, or set up their own ELK stack. AWS however also has their own service for this called CloudWatch Logs and CloudCaptain now has first class support for it!

At its core CloudWatch Logs is an API for ingesting and querying log events. It revolves around three simple concepts: Log Groups, Log Streams and Log Events.

Log Events

Log events are the actual log messages produced by your application. To be able to send those to CloudWatch Logs you need some kind of client to talk to the CloudWatch Logs API. By default AWS provides an agent to take care of that. It is unfortunately written in Python and comes with a long string of dependencies. In other words, it simply isn't a good fit for CloudCaptain's minimal images, so we had to come up with a better solution! More about this in a minute...

Log Streams

Log streams are the destination where log events get sent to and can be queried from.

With CloudCaptain, each application deployment within an environment receives its own log stream.

Log Groups

Log groups contain one or more log streams. Streams physically belong to a log group. Log events can also be queried across multiple streams at once at the log group level. And log event retention can also be configured for an entire group at once.

With CloudCaptain, each environment receives its own log group, containing all the logs streams of the applications deployed within that environment.

Seeing it in action

So let's see how this works in practice!

Creating an app with AWS CloudWatch Logs support

The first step is to create a new application configured for CloudWatch Logs:

We could also have done the same by using the CloudCaptain Client by creating an app with logs.type set to cloudwatch-logs:

> boxfuse create hello -logs.type=cloudwatch-logs

Fusing an image (including automatic agent installation and configuration)

As mentioned above, to be able to send events to AWS CloudWatch Logs a client is needed to talk to the API and unfortunately the official AWS agent doesn't cut it.

We are therefore today releasing a new open-source CloudWatch Logs agent written in Go that is optimized to work within CloudCaptain instances.

It has a very small footprint (3 MB) and is designed to asynchronously redirect stdout and stderr output of an application to CloudWatch Logs.

Now let's fuse an image of our app:

> boxfuse fuse hello.war

This will automatically install both our application and the CloudWatch Logs agent. Two copies of the agent will be running. One to redirect stdout output of the application to CloudWatch logs as INFO log events and the other to do the same for stderr output as ERROR log events.

Running an instance on VirtualBox

If we now run an instance of our application on VirtualBox using

> boxfuse run hello:1.0

CloudCaptain will also automatically launch a local CloudWatch Logs mock which will be ready to ingest log events as soon as the application starts sending them.

Running it on AWS

If we launch an instance on AWS with

> boxfuse run hello:1.0 -env=prod

...
Creating Log Group boxfuse/prod ...
Creating Log Stream boxfuse/prod > myuser/hello ...
...

You'll notice that both a Log Group for the CloudCaptain prod environment as well as a Log Stream for the myuser/hello app were created. The latter which the CloudCaptain CloudWatch Logs agent installed on your instance will now start sending log events to. Any stdout and stderr console output produced by our application will now be redirected to that log stream.

Retrieving the log events

To view the log events produced by our application all we need to do is

> boxfuse logs hello -env=prod

And you'll see the logs appearing in your terminal.

Live Tailing

But sometimes however it is also immensely useful to be able to follow logs in real time. And we are happy to announce that you can now do so both on VirtualBox and on AWS by simply appending -logs.tail to our prevent command:

> boxfuse logs hello -env=prod -logs.tail

Any new logs produced by your application will now immediately be shown on your machine for as long as the CloudCaptain Client is running. We have optimized things so that logs are delivered directly from AWS to your local machine, bypassing all CloudCaptain infrastructure.

Summary

CloudCaptain now has first-class support for AWS CloudWatch Logs. This gives you access to a centralized logging service which comes with durability, security, availability, searchability and speed.

With apps created to take advantage of this, your images are automatically fused with our brand new open-source CloudWatch Logs agent which is fully configured to redirect both stdout and stderr to AWS CloudWatch Logs.

CloudCaptain also automatically starts a local CloudWatch Logs mock to provide you with the same experience when running on VirtualBox.

And last but not least you now have easy access to your logs, including live tailing for both VirtualBox and AWS.

This new feature is available at no charge to all CloudCaptain users.

Deploy 7 MB Go VMs effortlessly to AWS

2016-08-09T00:00:00+00:00

Go is a highly productive language with a broad ecosystem for developing modern applications and microservices. It has the unique advantage of combining a great standard library with garbage collection and fast compilation down to a single native binary.

Today we're going to look at deploying Go applications effortlessly to AWS using CloudCaptain.

CloudCaptain is based upon 3 core principles:

1.	Immutable Infrastructure	Creating servers and never modifying them again by treating a server as one immutable unit that is regenerated after every change and promoted unchanged from environment to environment. This eliminates drift and increases reliability by ensuring you run the exact same code in production as the code you tested in test.
2.	Minimal Images	Analysing your application and generating minimal tailor-made Linux-based images on the fly that are 100x smaller than a typical Linux system and take just seconds to produce.
3.	Blue/Green deployments	Deploying a new version of an app in parallel to the existing one and only making the switch at the elastic IP or elastic load balancer level once the configured health checks of the new version have passed. Deployments are fully automated and effectively transactional, providing you with zero-downtime updates.

The plan

To see this in action, our plan is to first create a new Go web app which we'll fuse into a minimal CloudCaptain image that can be deployed unchanged both on VirtualBox and AWS.

We'll then deploy our image to VirtualBox (great for rapid feedback and quick local tests), push it to the CloudCaptain Vault (our secure online repository) and run it on AWS.

And finally we'll update it on AWS with zero-downtime blue/green deployments.

Prerequisites

Before we begin, ensure you have successfully:

created a CloudCaptain Account (simply log in with your GitHub account, it's free)
downloaded and installed the latest CloudCaptain Client
downloaded and installed the latest version of Go
downloaded and installed the latest version of VirtualBox

Creating the Go application

In this tutorial we are going to create and deploy a simple Go application.

Start by creating a directory for our app:

> mkdir getstarted-go

And navigate to it:

> cd getstarted-go

Now create a file called main.go with the following contents:

package main

import (
    "fmt"
    "net/http"
)

func handler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hi there, I love CloudCaptain!")
}

func main() {
    http.HandleFunc("/", handler)
    http.ListenAndServe(":8080", nil)
}

Now let's make sure we are compiling for Linux x64. If you are on Linux already, you can safely skip this step.

Windows users should invoke:

getstarted-go> set GOOS=linux

And Mac OSX users should invoke:

getstarted-go> export GOOS=linux

Now go ahead and compile the app into a single executable:

getstarted-go> go build -ldflags="-s"

Fusing a CloudCaptain image and running it locally on VirtualBox

Now it's time to fuse your application into a CloudCaptain image and launch an instance of it on VirtualBox:

getstarted-go> boxfuse run -ports.http=8080 -image=getstarted-go:1.0

Creating getstarted-go ...
Mapping getstartedgo-dev-myuser.boxfuse.io to 127.0.0.1 ...
Created App getstarted-go (single-instance / none)
Fusing Image for getstarted-go (ELF64) ...
Image fused in 00:00.517s (7170 K) -> myuser/getstarted-go:1.0
Launching Instance of myuser/getstarted-go:1.0 on VirtualBox ...
Forwarding http port localhost:8080 -> vb-f6dba4e5:8080
Instance launched in 00:03.055s -> vb-f6dba4e5
Waiting for Payload to start on vb-f6dba4e5:8080 (expecting HTTP 200 at / within 300s) ...
Payload started in 00:00.009s -> https://127.0.0.1:8080

In just a few seconds CloudCaptain found your application, detected its type, generated an image for it and launched an instance of that image on VirtualBox.

Now open your browser and navigate to this address to see your new application up and running within the VirtualBox VM:

You can also see your newly created image:

getstarted-go> boxfuse ls

Images available locally:
+--------------------------+---------------+-------+---------+--------------+--------+---------------------+
| Image                    |    Payload    | Debug | Runtime |    Ports     |  Size  |    Generated at     |
+--------------------------+---------------+-------+---------+--------------+--------+---------------------+
| myuser/getstarted-go:1.0 | getstarted-go | false | ELF64   | http -> 8080 | 7170 K | 2016-08-08 19:38:56 |
+--------------------------+---------------+-------+---------+--------------+--------+---------------------+
Total: 1

As well as the instance that is running:

getstarted-go> boxfuse ps

Running Instances on VirtualBox in the dev environment :
+-------------+--------------------------+---------------------+-----------------------+---------------------+
|  Instance   |          Image           |        Type         |          URL          |     Launched at     |
+-------------+--------------------------+---------------------+-----------------------+---------------------+
| vb-f6dba4e5 | myuser/getstarted-go:1.0 | 4 CPU / 1024 MB RAM | https://127.0.0.1:8080 | 2016-08-08 19:38:59 |
+-------------+--------------------------+---------------------+-----------------------+---------------------+
Total: 1

Deploying your application to AWS

Now let's deploy the image to AWS. As CloudCaptain works with your AWS account, it first needs the necessary permissions to do so. So if you haven't already done it, go to the CloudCaptain Console and connect your AWS account now.

Every new CloudCaptain account comes with 3 environments: dev, test and prod. dev is your local VirtualBox environment and test and prod are on AWS.

So let's deploy our application to the prod environment on AWS:

getstarted-go> boxfuse run getstarted-go:1.0 -env=prod

Pushing myuser/getstarted-go:1.0 ...
Verifying myuser/getstarted-go:1.0 ...
Waiting for AWS to create an AMI for myuser/getstarted-go:1.0 in eu-central-1 (this may take up to 50 seconds) ...
AMI created in 00:29.835s in eu-central-1 -> ami-9cf006f3
Creating Elastic IP ...
Mapping getstartedgo-myuser.boxfuse.io to 52.28.244.5 ...
Creating security group boxsg-myuser-prod-getstarted-go-1.0 ...
Launching t2.micro instance of myuser/getstarted-go:1.0 (ami-9cf006f3) in prod (eu-central-1) ...
Instance launched in 00:17.580s -> i-921e9a2f
Creating Cloud Watch Alarm for Instance auto-recovery -> i-921e9a2f-auto-recovery-alarm
Waiting for AWS to boot Instance i-921e9a2f and Payload to start at https://54.93.51.85:8080/ ...
Payload started in 00:28.022s -> https://54.93.51.85:8080/
Remapping Elastic IP 52.28.244.5 to i-921e9a2f ...
Waiting 15s for AWS to complete Elastic IP Zero Downtime transition ...
Deployment completed successfully. myuser/getstarted-go:1.0 is up and running at https://getstartedgo-myuser.boxfuse.io:8080/

Notice that we have now specified an image, as we want to reuse our image unchanged instead fusing a new one.

With that one command CloudCaptain has automatically pushed your image to the CloudCaptain Vault as well as provisioned, configured and secured all necessary AWS resources. There is no manual work necessary on your behalf.

All you need to do is simply navigate to your new domain to see your Go application in action on AWS:

Bonus: update your application using blue/green deployments

Now let's take things one step further and deploy an update of your application with zero downtime.

Start by modifying main.go with a simple change:

...
fmt.Fprintf(w, "Hi there, I love CloudCaptain!")
fmt.Fprintf(w, "Updated by CloudCaptain with zero downtime!")
...

Then rebuild the tgz:

getstarted-go> go build -ldflags="-s"

Finally, deploy the new version of your application to AWS:

getstarted-go> boxfuse run -ports.http=8080 -env=prod -image=getstarted-go:2.0

Fusing Image for getstarted-go (ELF64) ...
Image fused in 00:00.541s (7170 K) -> myuser/getstarted-go:2.0
Pushing myuser/getstarted-go:2.0 ...
Verifying myuser/getstarted-go:2.0 ...
Waiting for AWS to create an AMI for myuser/getstarted-go:2.0 in eu-central-1 (this may take up to 50 seconds) ...
AMI created in 00:14.649s in eu-central-1 -> ami-ba8b7dd5
Creating security group boxsg-myuser-prod-getstarted-go-2.0 ...
Launching t2.micro instance of myuser/getstarted-go:2.0 (ami-ba8b7dd5) in prod (eu-central-1) ...
Instance launched in 00:19.492s -> i-d91f9b64
Creating Cloud Watch Alarm for Instance auto-recovery -> i-d91f9b64-auto-recovery-alarm
Waiting for AWS to boot Instance i-d91f9b64 and Payload to start at https://52.59.244.104:8080/ ...
Payload started in 00:30.181s -> https://52.59.244.104:8080/
Remapping Elastic IP 52.28.244.5 to i-d91f9b64 ...
Waiting 15s for AWS to complete Elastic IP Zero Downtime transition ...
Destroying Cloud Watch Alarm i-921e9a2f-auto-recovery-alarm ...
Terminating instance i-921e9a2f ...
Destroying Security Group sg-dbc040b3 (boxsg-myuser-prod-getstarted-go-1.0) ...
Deployment completed successfully. myuser/getstarted-go:2.0 is up and running at https://getstartedgo-myuser.boxfuse.io:8080/

And there it is:

Summary

In this post, we saw how to deploy and update a Go application to AWS using CloudCaptain in 3 easy steps.

First we fused our application into a 7 MB CloudCaptain image in under 1 second and ran an instance of it on VirtualBox. We then deployed the same image unchanged to AWS. And finally we updated our application on AWS with zero downtime using blue/green deployments.

To do so we used CloudCaptain and its 3 core principles: Immutable Infrastructure , Minimal Images and Blue/Green deployments.

And if you are a Revel fan, CloudCaptain has you covered with first-class support for running Revel applications!

However this is just the beginning! CloudCaptain also supports dead-easy auto-scaling, database auto-provisioning and much more. Not only for Go apps, but also for Node.js and JVM ones.

So if you haven't already, sign up for your CloudCaptain account now (simply log in with your GitHub id, it's free) and start deploying your Go applications effortlessly to AWS in minutes.

Worker Apps

2016-08-04T00:00:00+00:00

Until now CloudCaptain came with support for two types of applications: single-instance and load-balanced.

single-instance applications sit behind an Elastic IP address and are updated with zero-downtime blue/green deployments. Even though they lack runtime redundancy, they are great for non-critical workloads typically involving HTTP or HTTPS requests.

load-balanced applications on the other hand sit behind an Elastic Load Balancer. Not only are they also updated with zero-downtime blue/green deployments, but they also come with runtime-redundancy, zero-downtime auto-recovery and auto-scaling. They are a great choice for robust and demanding HTTP and HTTPS workloads.

While single-instance and load-balanced apps serve quite a lot of needs, there are times when you simply don't need a stable entry point in your application. For those scenarios having an extra Elastic IP or ELB simply adds unnecessary complexity and costs.

To address this, today we are introducing a third kind of application: worker apps.

Introducing Worker Apps

worker apps come with the same reliability advantages as load-balanced apps. They are updated with zero-downtime blue/green deployments and they come with runtime-redundancy, zero-downtime auto-recovery and auto-scaling. However, they don't have an Elastic Load Balancer in front of them.

Let's look at 3 scenarios where they are particularly useful.

Client-side load balancing

When Netflix decided to go all-in on AWS, it was still early days for the cloud. At that time ELBs could only be internet facing, and simply weren't a great choice for internal communication between services like they are today.

Those needs sparked the creation of a number of open-source tools to address them including Eureka and Ribbon. You can think of Eureka as a central place where instances services can register themselves with, so they can be discovered by clients. Ribbon is a library that you run on the client that will then take care of load balancing calls between multiple instances of the same service that were discovered via Eureka.

To make it easier to run setups like the one above locally, we have also made changes to the way we handle networking with VirtualBox. With VirtualBox 5.1 and newer all instances in the dev environment will now receive their own unique IP address with a dedicated NAT network and will be able to talk directly to each other.

Queues

Another common scenario for worker apps is queueing. In this case services communicate asynchronously with each other via queues. These could be SQS queues or any other async solution. worker apps can be great message consumers and their autoscaling capabilities can be a great fit for dealing with varying loads.

Cron jobs

Finally we also have periodic tasks, or cron jobs, where the trigger is time based. Even if it is just a single instance running, the auto-scaling group of a worker app will comes in handy to automatically restart a new one should the old one die.

Summary

With worker apps, you now have a third kind of application available. Unlike single-instance and load-balanced apps they do not have an Elastic IP or ELB as an entry point as this is simply not needed for a number of scenarios including client-side load balancing, queue consumers and cron job-driven apps. In those cases worker apps reduce both cost and complexity.

So go ahead and launch a new worker app today! And if you haven't signed up already, simply log in with your GitHub id (it's free!) to start deploying your JVM and Node.js applications effortlessly on AWS in minutes!

Live Reloading of Spring Boot Apps on AWS with DevTools

2016-07-22T00:00:00+00:00

In this post we're going to take a look at one of Spring Boot's best kept secrets: the Spring Boot DevTools.

As great as it is to test things out on your local machine, there are also clear benefits to being able to both develop and debug your application in an environment that is as similar as possible to its production environment. And that's where the Spring Boot DevTools come in. They let you live reload and remote debug your application no matter where it is running.

So let's dive in and see in 3 easy steps how you set up the Spring Boot DevTools in minutes to live reload a Spring Boot app fully automatically on AWS.

How do the Spring Boot DevTools work?

You can think of the SpringBoot DevTools as a set of components focused on boosting developer productivity.

They consist of 3 major parts:

The RemoteApplication runner that sends any changed class files and resources from your IDE to the running instance
The DevTools jar inside your application that interacts with the RemoteApplication to live reload changes
A LiveReload Server in your IDE that triggers the LiveReload browser plugin to automatically reload the current page

Together they provide you with super fast way to iteratively develop your application while it is running in a production-like environment.

Step 0: Settings things up

In order to demonstrate how the DevTools work, we're going to need an application to work with. Let's create one in a few easy steps.

Generating the skeleton

First we'll need the basic project structure. Head to start.spring.io and fill in the group and artifact id:

Then select both Web and DevTools dependencies and click Generate Project.

Adding a controller

For our application to do anything useful we'll need to add a controller.

Now unpack the downloaded archive and add the Controller method and annotations to the devtools-demo/src/main/java/demo/DevtoolsDemoApplication.java file:

package demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.bind.annotation.RestController;

import java.util.Date;

@SpringBootApplication
@RestController
public class DevtoolsDemoApplication {
    @RequestMapping(method = RequestMethod.GET, path = "/")
    @ResponseBody
    public String currentTime() {
        return "The current time is " + new Date();
    }

	public static void main(String[] args) {
		SpringApplication.run(DevtoolsDemoApplication.class, args);
	}
}

This is just a very simple controller that will print out the current time when it receives an HTTP GET request at /.

Setting the remote secret

In order for your application to ensure only authorized users trigger live reloading you must also set the remote secret in devtools-demo/src/main/resources/application.properties:

spring.devtools.remote.secret=mysecret

Building the application

Now that we have a controller and a secret set up, it is time to build the application.

First set the version to 0.0.1 in devtools-demo/pom.xml:

<version>0.0.1</version>

And ensure the Spring Boot Dev Tools are enabled in the final artifact by setting the excludeDevtools flag of the Spring Boot Maven plugin to false:

<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
    <configuration>
        <excludeDevtools>false</excludeDevtools>
    </configuration>
</plugin>

Then open a terminal window and navigate to the directory where you extracted the archive. Go inside the newly created devtools-demo directory and invoke Maven to create an executable jar:

devtools-demo> mvnw package

After a few seconds you will then have a devtools-demo-0.0.1.jar file inside the target directory.

Step 1: Deploying the application to AWS

Now that we have things set up, it is time to deploy the application to AWS. Feel free to use the tool of your choice.

If you choose to use CloudCaptain all you need to do is literally:

devtools-demo> boxfuse run -env=test

And everything will be set up, configured and secured automatically for you:

Creating devtools-demo ...
Mapping devtoolsdemo-dev-myuser.boxfuse.io to 127.0.0.1 ...
Created App devtools-demo (single-instance / none)
Fusing Image for devtools-demo-0.0.1.jar ...
Image fused in 00:04.168s (56336 K) -> myuser/devtools-demo:0.0.1
Pushing myuser/devtools-demo:0.0.1 ...
Verifying myuser/devtools-demo:0.0.1 ...
Waiting for AWS to create an AMI for myuser/devtools-demo:0.0.1 in eu-central-1 (this may take up to 50 seconds) ...
AMI created in 00:31.724s in eu-central-1 -> ami-ed936782
Creating Elastic IP ...
Mapping devtoolsdemo-myuser.boxfuse.io to 52.59.61.93 ...
Creating security group boxsg-myuser-devtools-demo-0.0.1 ...
Launching t2.micro instance of myuser/devtools-demo:0.0.1 (ami-ed936782) in prod (eu-central-1) ...
Instance launched in 00:24.954s -> i-ed279f50
Creating Cloud Watch Alarm for Instance auto-recovery -> i-ed279f50-auto-recovery-alarm
Waiting for AWS to boot Instance i-ed279f50 and Payload to start at https://52.59.240.195:8080/ ...
Payload started in 00:46.229s -> https://52.59.240.195:8080/
Remapping Elastic IP 52.59.61.93 to i-ed279f50 ...
Waiting 15s for AWS to complete Elastic IP Zero Downtime transition ...
Deployment completed successfully. myuser/devtools-demo:0.0.1 is up and running at https://devtoolsdemo-myuser.boxfuse.io:8080/

Make sure your application is up and running correctly on AWS:

Step 2: Live reloading classes and resources

Now that everything is up and running, let's set up live reloading. As described in the diagram above there are two types of reloading we must set up for everything to work smoothly:

automatic reloading of classes and resources inside our AWS instance
automatic reloading of the page in the browser

Let's focus on reloading of classes and resources first:

For reloading to be triggered whenever something changes in our IDE, we must start a local process to talk to our AWS instance. While this sounds complicated, Spring Boot makes this absolutely trivial. Go to your IDE and create a new run configuration for our devtools-demo project. And now instead of running our application itself, we'll simply run org.springframework.boot.devtools.RemoteSpringApplication with one argument: the address where our application is deployed.

(IntelliJ users can safely ignore the bogus Run Configuration Error)

You should now see the RemoteApplication starting up and connecting with your instance:

  .   ____          _                                              __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _          ___               _      \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` |        | _ \___ _ __  ___| |_ ___ \ \ \ \
 \\/  ___)| |_)| | | | | || (_| []::::::[]   / -_) '  \/ _ \  _/ -_) ) ) ) )
  '  |____| .__|_| |_|_| |_\__, |        |_|_\___|_|_|_\___/\__\___|/ / / /
 =========|_|==============|___/===================================/_/_/_/
 :: Spring Boot Remote ::  (v1.4.0.RC1)

2016-07-21 17:35:42.861  INFO 4828 --- [           main] o.s.b.devtools.RemoteSpringApplication   : Starting RemoteSpringApplication v1.4.0.RC1 on axel-silencio with PID 4828
2016-07-21 17:35:42.861  INFO 4828 --- [           main] o.s.b.devtools.RemoteSpringApplication   : No active profile set, falling back to default profiles: default
2016-07-21 17:35:42.893  INFO 4828 --- [           main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@16ec5519: startup date [Thu Jul 21 17:35:42 CEST 2016]; root of context hierarchy
2016-07-21 17:35:43.209  WARN 4828 --- [           main] o.s.b.d.r.c.RemoteClientConfiguration    : The connection to https://devtoolsdemo-myuser.boxfuse.io:8080 is insecure. You should use a URL starting with 'https://'.
2016-07-21 17:35:43.262  INFO 4828 --- [           main] o.s.b.d.a.OptionalLiveReloadServer       : LiveReload server is running on port 35729
2016-07-21 17:35:43.278  INFO 4828 --- [           main] o.s.b.devtools.RemoteSpringApplication   : Started RemoteSpringApplication in 0.755 seconds (JVM running for 1.058)

Now change the currentTime() method to:

@RequestMapping(method = RequestMethod.GET, path = "/")
@ResponseBody
public String currentTime() {
    return "The current time is " + new Date()
            + " and here is a random number: " + new Random().nextInt();
}

After saving and compiling in your IDE you'll see the following line in the output of the RemoteApplication:

2016-07-21 17:36:10.119  INFO 3268 --- [pool-1-thread-1] o.s.b.d.r.c.DelayedLiveReloadTrigger     : Remote server has changed, triggering LiveReload

And you'll see the changes in your browser after refreshing the page:

Step 3: Automatic reloading of the page in the browser

The setup we now have will instantly reload any changes to our application. It does however still require us to manually refresh the browser window to see the changes. Let's fix that!

We're going to do this using LiveReload. There are two parts to this: a browser extension that reloads the current page and a server that notifies it when it should do so.

As some of you may have noticed from the output when you started the RemoteApplication, the Spring Boot DevTools actually automatically start a LiveReload server by default:

2016-07-21 17:35:43.262  INFO 4828 --- [           main] o.s.b.d.a.OptionalLiveReloadServer       : LiveReload server is running on port 35729

This means that all we actually need now is to download and install the LiveReload browser extension.

Once you have done that, activate the extension with a click:

And if you now change the currentTime() method to:

@RequestMapping(method = RequestMethod.GET, path = "/")
@ResponseBody
public String currentTime() {
    return "It's live reloading time!";
}

After you save and compile, the browser pointed at your Spring Boot app running on AWS will now automatically refresh to show the changes:

Summary

It is great to be able to iterate quickly in a production-like environment. We saw how easy it is to do so using one of Spring Boot's best kept secrets: the Spring Boot DevTools.

They bring the productivity of local development directly to remote deployments running on AWS or the environment of your choice. And if you add the LiveReload browser extension you won't even need to refresh to see the changes!

So go ahead, set the DevTools up, have fun and start Live Reloading your Spring Boot apps on AWS in minutes!

Shameless plug: To deploy your Spring Boot applications effortlessly to AWS, simply log in to CloudCaptain with your GitHub account and you'll be up and running in no time. With just a single command all required AWS resources will be provisioned, configured and secured for you, and all updates are performed as zero-downtime blue/green deployments using Immutable Infrastructure.

Live Reloading

2016-07-05T00:00:00+00:00

At CloudCaptain we strongly believe in optimizing for developer productivity. This means easy of use, robustness and performance. This is the reason we worked tirelessly to bring you 2 second image generation for Node.js apps, dead easy debugging and profiling and more.

Today we are taking things further by introducing support for Live Reloading in the dev environment.

This is all about reducing cycle time by enabling super-fast turnarounds. When you are in the middle of developing something you want to be able to experiment and see results as fast as possible. To enable this, we are introducing a new type of image called live images. They can be fused from a directory instead of an archive. This directory will usually be the output directory of your IDE or your general project directory. Instances launched from live images automatically keep track of the changes you make to that directory and ensure that those changes are now visible immediately, without the need to go through a full kill/fuse/run cycle.

How do Live Images work?

Live images achieve their magic by combining two techniques:

mounting your application files directory as a live directory within your instance
adding a reloading agent to your instance to reload your application when needed

For JVM apps, CloudCaptain supports mounting exploded jars, exploded wars and exploded Play zips directly into your instance. Additionally CloudCaptain automatically configures your JVM to use the Spring Loaded Java Agent for automatically reloading classes when the underlying class file changes.

For Node.js apps, CloudCaptain supports mounting your Node.js application directory (the one containing package.json) directly into your instance. Additionally, CloudCaptain automatically starts your Node.js app with node-supervisor to ensure it is properly restarted whenever js files change.

Seeing it in Action

Let's have a look at how this works in practice. For this we're going to use a simple Node.js app using Express.

Start by installing Express:

> npm install -g express-generator

Then create the app:

> express live-reloading-test

And navigate to its directory:

> cd live-reloading-test

Ensure all dependent modules have been downloaded locally:

> npm install

Then fuse and run a live image of the directory by using -live:

> boxfuse run -live

...
Fusing Image for live-reloading-test ...
Image fused in 00:01.290s (15242 K) -> myuser/live-reloading-test:0.0.0
Launching Instance of myuser/live-reloading-test:0.0.0 on VirtualBox ...
Mounting Live directory C:\Workspaces\live-reloading-test into vb-2656d939 ...
Forwarding http port localhost:80 -> vb-2656d939:80
Instance launched in 00:03.476s -> vb-2656d939
Waiting for Payload to start on Instance vb-2656d939 ...
...
vb-2656d939 => Launching App ...
vb-2656d939 => node-supervisor -t -s -k ./bin/www
vb-2656d939 =>
vb-2656d939 => Running node-supervisor with
vb-2656d939 =>   program './bin/www'
vb-2656d939 =>   --watch '.'
vb-2656d939 =>   --non-interactive
vb-2656d939 =>   --extensions 'node,js,/bin/www'
vb-2656d939 =>   --exec 'node'
vb-2656d939 =>
vb-2656d939 => Starting child process with 'node ./bin/www'
vb-2656d939 => Watching directory '/live' for changes.
Payload started in 00:02.395s -> https://127.0.0.1

You can now see your new live instance up and running:

Now go ahead and modify routes/index.js by changing the title:

var express = require('express');
var router = express.Router();

/* GET home page. */
router.get('/', function(req, res, next) {
  res.render('index', { title: 'Express' });
  res.render('index', { title: 'CloudCaptain Live Reloading' });
});

module.exports = router;

As soon as you save the file you'll see this in the instance logs:

vb-2656d939 => crashing child with SIGKILL
vb-2656d939 => Thu Jul 07 2016 07:11:54 GMT+0000 (UTC)
vb-2656d939 => Starting child process with 'node ./bin/www'

Finally simply refresh the browser to see the changes instantly without fusing a new image:

Summary

With Live Reloading you can now have super fast roundtrips by making changes to your project directory and seeing them immediately within your instance. This is a huge boost for productivity in development and it is available now on Windows, Mac OSX and Linux.

Existing CloudCaptain users can simply update their client with boxfuse -u and enjoy!

And if you haven't signed up already, simply log in with your GitHub id (it's free!) to start deploying your JVM and Node.js applications effortlessly on AWS in minutes and live reloading them on VirtualBox in seconds!

Immutable Infrastructure Talk and Interview at QCon London 2016

2016-06-08T00:00:00+00:00

Both last year and this year there have been a large number of CloudCaptain events with many more to come!

In March, CloudCaptain founder and CEO, Axel Fontaine spoke about Immutable Infrastructure: Rise of the Machine Images at QCon London 2016. This talk details the principles and the lessons learned CloudCaptain is based upon. And it was a huge success with over 400 attendees in the room who gave it excellent reviews.

What attendees had to say about it

Immutable Infrastructure talk by @axelfontaine has been brilliant at #qconlondon
— Jamie Kelly (@jk563) March 7, 2016

#qconlondon. Great talk by @axelfontaine. Nice to see a live demo that shows real life technical solutions.
— Martin Peston (@phuturespace) March 7, 2016

Rise of the Machines Images: Very nice talk about immutable infrastructure by @axelfontaine #qconlondon pic.twitter.com/2Ujc0OcCSr
— Julien Vey (@julienvey) March 7, 2016

Video and Slides of the talk

In case you didn't have a chance to attend the talk live, you watch a video recording of it and download the slides on InfoQ.

Interview (with full transcript)

Axel Fontaine also sat down with João Miranda for an interview to discuss the topic in additional detail:

Report

Finally Manuel Pais from InfoQ also wrote a report of the talk for you to enjoy!

Have fun!

Have fun watching the talk and the interview as well as reading the report! And don't forget to come and say hi at one of our future events!

Oracle JRE and Kernel Tuning

2016-06-02T00:00:00+00:00

CloudCaptain's goal is to make you productive by making deployments of your Node.js and JVM-based applications truly effortless. And that's why we worked tirelessly to make it possible to deploy most applications out-of-the-box by issuing just a single boxfuse run command that uses convention-over-configuration with sensible defaults.

Some applications however require finer grained control, which is why we introduced customizing temp space allocation, restricting ports, dead-easy debugging and profiling and more.

Today we are introducing two new features to give you even more control: custom JREs and kernel tuning.

Custom JREs

By default CloudCaptain uses OpenJDK for JVM-based applications, with the possibility to opt for either the OpenJDK 7 or 8 headless JREs. This is a great open-source choice for the vast majority of the applications, both from a technical as well as from a legal perspective.

Some users however require commercial support or features only available with the Oracle JRE. Due to Oracle licensing restrictions we unfortunately cannot offer it as a component from the CloudCaptain Inventory as the Oracle JRE license explicitly prohibits standalone redistribution of the Oracle JRE.

However, starting today CloudCaptain now supports custom JREs. You can now bundle any Linux x64 Java Runtime Environment of your choice, including the Oracle JRE, directly with your application. CloudCaptain will then automatically use that JRE instead of the default OpenJDK one.

Using the Oracle JRE

Here is a small example of how to use the Oracle JRE with a Spring Boot application:

1. Download the Linux x64 JRE of your choice directly from the Oracle website.

2. Extract the /jre directory from the tar.gz distribution and add it to the src/main/resources folder of your application:

 my-spring-boot-app
   src
     main
       java
       resources
         jre
   bin
     java
     ...
   lib
     amd64
     ...
     rt.jar
     ...
   COPYRIGHT
   LICENSE
   ...

3. Package your application:

my-spring-boot-app> mvn package

4. Fuse a CloudCaptain image:

my-spring-boot-app> boxfuse fuse

5. Done! Your application now uses the Oracle JRE instead of the OpenJDK one when launched from that image.

Kernel Tuning

The other feature we are introducing today is kernel tuning. While most users will never need this, this is advanced feature can be a life-saver for expert users who require low-level tuning of the Linux kernel.

All you need for this is simply to ship a sysctl.conf file like the one below with their application:

net.ipv4.tcp_fastopen = 1
net.ipv4.tcp_fastopen_key = 00000000-00000000-00000000-00000000
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_frto = 2
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_limit_output_bytes = 131072

CloudCaptain will then automatically apply these settings on instance startup.

Conclusion

CloudCaptain is all about making you productive by making deployment of your apps truly effortless. With custom JREs and kernel tuning CloudCaptain now gives advanced users additional ways to customize their images.

Custom JREs now also allow you to easily run your applications using the Oracle JRE instead of OpenJDK and kernel tuning lets expert users to perform low-level tuning of the Linux kernel inside their images.

Existing CloudCaptain users can simply update with

> boxfuse -u

to use these features. Enjoy!

So if you haven't already, sign up for your CloudCaptain account now (simply log in with your GitHub id, it's free) to start deploying your JVM and Node.js applications effortlessly on AWS in minutes.

One Year of CloudCaptain

2016-05-25T00:00:00+00:00

Last month marked one year since the initial version of CloudCaptain was released. We are now many many features and over 100 CloudCaptain Client releases further. And what a ride it's been!

CloudCaptain set out to make the deployment of applications to the cloud absolutely effortless by leveraging its three core principles:

1.	Immutable Infrastructure	Creating servers and never modifying them again by treating a server as one immutable unit that is regenerated after every change and promoted unchanged from environment to environment. This eliminates drift and increases reliability by ensuring you run the exact same code in production as the code you tested in test.
2.	Minimal Images	Analysing your application and generating minimal tailor-made Linux-based images on the fly that are 100x smaller than a typical Linux system and take just seconds to produce.
3.	Blue/Green deployments	Deploying a new version of an app in parallel to the existing one and only making the switch at the elastic IP or elastic load balancer level once the configured health checks of the new version have passed. Deployments are fully automated and effectively transactional, providing you with zero-downtime updates.

The humble beginnings

We had to start out somewhere and so we picked one problem and decided to solve it better than anyone else out there: deploying Dropwizard applications to AWS:

This provided us with a great testbed to both validate our innovative technology (generating minimal images on-the-fly in seconds) as well as to pioneer our deep integration features (extracting port and healthcheck configuration directly out of Dropwizard's native configuration files)

We launched with support for testing locally on VirtualBox and performing blue/green deployments atomically on AWS with zero downtime updates. We introduced this both for single instance applications (using an Elastic IP) as well as load balanced applications (using an Elastic Load Balancer).

We have always been strong proponents of Continuous Deployments and making CloudCaptain a natural fit for this was a no-brainer. From day one we shipped both a cross-platform CLI (the CloudCaptain Command-line Client) as well as a build tool plugin (the CloudCaptain Maven plugin) to make it trivial to automate your deployment workflows.

Expanding to more JVM frameworks

Building upon the foundation laid out with Dropwizard, CloudCaptain successively added deep integration for most common modern JVM application frameworks. Over the course of this first year, CloudCaptain added support for Spring Boot, Play, Grails, plain executable Jars, Tomcat and even TomEE for JavaEE applications.

We also made sure it is easy for users of those frameworks to discover CloudCaptain by contributing sections to the official documentation of both Spring Boot and the Play framework.

And we introduced the CloudCaptain Gradle Plugin (available directly from the Gradle Plugin Portal) to make the integration even nicer.

Auto-scaling

Auto-scaling is one of those true innovations of the cloud and it fits CloudCaptain's immutable minimal images like a glove. So it was only natural that we added first-class support to dynamically scale the capacity of your system based on either CPU or network usage:

Database auto-provisioning

As many users rely on relational databases to store their data, CloudCaptain stepped in to make sure it became absolutely trivial to provision PostgreSQL and MySQL databases reliably across all environments from dev on VirtualBox to test and prod on AWS.

This also marked the introduction of the CloudCaptain Dev VM on VirtualBox, providing you with fully configured database instances ready to be used by your application, without the need to install those yourself on your machine.

Wherever your framework of choices supports it, CloudCaptain also automatically configures it to automatically use the correct database instance with the correct credentials in each environment.

And of course to easily and reliably evolve your database schema over time, our database migration tool Flyway is there to help you too.

Debugging and Profiling

To make it easier to see what's going on with your application, CloudCaptain introduced first class support for debugging and profiling, both locally on VirtualBox and directly in the cloud for your instances running on AWS.

As many of those protocols are insecure, CloudCaptain automatically makes use of its support for restricted ports to ensure that by default those interfaces are only accessible from your current IP.

Node.js

Based on widespread demand from our customers, we added support for our first language platform beyond the JVM with first class Node.js support.

As the Node.js runtime is more compact this resulted in super fast fusing times with images as small as 15 MB!

This is just the beginning

To mark the beginning of our second year, we are kicking things off with a bang and have released support for three new features:

Node.js LTS releases: you can now use all current Node.js releases with CloudCaptain in addition to the latest one
Instance auto-recovery: all single instance applications now automatically recover to new hardware with no manual intervention required if the current underlying EC2 hardware becomes impaired
Subnets restrictions: load balanced applications can now restrict exactly which subnets they wish to use within a VPC

Enjoy!

But remember, this is just the beginning. We are only getting started!

Deploying 15 MB Node.js VMs effortlessly to AWS

2016-03-23T00:00:00+00:00

Node.js is a highly productive platform with a broad ecosystem for developing modern web applications and microservices.

Today we're going to look at deploying Node.js applications effortlessly to AWS using CloudCaptain.

CloudCaptain is based upon 3 core principles:

1.	Immutable Infrastructure	Creating servers and never modifying them again by treating a server as one immutable unit that is regenerated after every change and promoted unchanged from environment to environment. This eliminates drift and increases reliability by ensuring you run the exact same code in production as the code you tested in test.
2.	Minimal Images	Analysing your application and generating minimal tailor-made Linux-based images on the fly that are 100x smaller than a typical Linux system and take just seconds to produce.
3.	Blue/Green deployments	Deploying a new version of an app in parallel to the existing one and only making the switch at the elastic IP or elastic load balancer level once the configured health checks of the new version have passed. Deployments are fully automated and effectively transactional, providing you with zero-downtime updates.

The plan

Our plan is to first create a new Node.js app which we'll fuse into a minimal CloudCaptain image that can be deployed unchanged both on VirtualBox and AWS.

We'll then deploy our image to VirtualBox (great for rapid feedback and quick local tests), push it to the CloudCaptain Vault (our secure online repository) and run it on AWS.

And finally we'll update it on AWS with zero-downtime blue/green deployments.

Prerequisites

Before we begin, ensure you have successfully:

created a CloudCaptain Account (simply log in with your GitHub account, it's free)
downloaded and installed the latest CloudCaptain Client
downloaded and installed the latest version of Node.js
downloaded and installed the latest version of VirtualBox

Creating the Node.js application

In this tutorial we are going to create and deploy a simple Node.js application based on Express.

Start by installing the necessary npm packages:

> npm install -g express-generator npm-bundle

Now generate the project:

> express getstarted-nodejs

And navigate to the newly created directory:

> cd getstarted-nodejs

Then simply create the bundle tgz in the directory based on your package.json:

getstarted-nodejs> npm-bundle

Fusing a CloudCaptain image and running it locally on VirtualBox

Now it's time to fuse your application into a CloudCaptain image and launch an instance of it on VirtualBox:

getstarted-nodejs> boxfuse run

Fusing Image for getstarted-nodejs-0.0.0.tgz ...
Image fused in 00:02.481s (15076 K) -> myuser/getstarted-nodejs:0.0.0
Launching Instance of myuser/getstarted-nodejs:0.0.0 on VirtualBox ...
Forwarding http port localhost:80 -> vb-b18d6746:80
Instance launched in 00:03.139s -> vb-b18d6746
Waiting for Payload to start on Instance vb-b18d6746 ...
Payload started in 00:04.688s -> https://127.0.0.1

In just a few seconds CloudCaptain found your application, detected its type, generated an image for it and launched an instance of that image on VirtualBox.

Now open your browser and navigate to this address to see your new application up and running within the VirtualBox VM:

You can also see your newly created image:

getstarted-nodejs> boxfuse ls

Images available locally:
+--------------------------------+-----------------------------+-------+---------------+------------+---------+---------------------+
| Image                          |           Payload           | Debug |    Runtime    |   Ports    |  Size   |    Generated at     |
+--------------------------------+-----------------------------+-------+---------------+------------+---------+---------------------+
| myuser/getstarted-nodejs:0.0.0 | getstarted-nodejs-0.0.0.tgz | false | Node.js 5.8.0 | http -> 80 | 15076 K | 2016-03-12 14:57:50 |
+--------------------------------+-----------------------------+-------+---------------+------------+---------+---------------------+
Total: 1

As well as the instance that is running:

getstarted-nodejs> boxfuse ps

Running Instances on VirtualBox in the dev environment :
+-------------+--------------------------------+---------------------+------------------+---------------------+
|  Instance   |             Image              |        Type         |        URL       |     Launched at     |
+-------------+--------------------------------+---------------------+------------------+---------------------+
| vb-b18d6746 | myuser/getstarted-nodejs:0.0.0 | 4 CPU / 1024 MB RAM | https://127.0.0.1 | 2016-03-12 14:57:56 |
+-------------+--------------------------------+---------------------+------------------+---------------------+
Total: 1

Deploying your application to AWS

Every new CloudCaptain account comes with 3 environments: dev, test and prod. dev is your local VirtualBox environment and test and prod are on AWS.

So let's deploy our application to the prod environment on AWS:

getstarted-nodejs> boxfuse run -env=prod

Creating myuser/getstarted-nodejs ...
Pushing myuser/getstarted-nodejs:0.0.0 ...
Verifying myuser/getstarted-nodejs:0.0.0 ...
Waiting for AWS to create an AMI for myuser/getstarted-nodejs:0.0.0 in eu-central-1 (this may take up to 50 seconds) ...
AMI created in 00:17.463s in eu-central-1 -> ami-d98364b6
Creating Elastic IP ...
Mapping getstartednodejs-myuser.boxfuse.io to 52.58.12.244 ...
Creating security group boxsg-myuser-prod-getstarted-nodejs-0.0.0 ...
Launching t2.micro instance of myuser/getstarted-nodejs:0.0.0 (ami-d98364b6) in prod (eu-central-1) ...
Instance launched in 00:40.942s -> i-af499712
Waiting for AWS to boot Instance i-af499712 and Payload to start at https://54.93.101.157/ ...
Payload started in 00:10.313s -> https://54.93.101.157/
Remapping Elastic IP 52.58.12.244 to i-af499712 ...
Waiting 15s for AWS to complete Elastic IP Zero Downtime transition ...
Deployment completed successfully. myuser/getstarted-nodejs:0.0.0 is up and running at https://getstartednodejs-myuser.boxfuse.io/

Notice that we have now specified an image, as we want to reuse our image unchanged instead fusing a new one.

All you need to do is simply navigate to your new domain to see your Node.js application in action on AWS:

Bonus: update your application using blue/green deployments

Now let's take things one step further and deploy an update of your application with zero downtime.

Start by modifying views/index.jade with a simple change:

extends layout

block content
  h1= title
  p Updated by CloudCaptain with zero downtime!

then bump the version in package.json:

"version": "0.0.1",

clean the old bundle:

getstarted-nodejs> del *.tgz

and rebuild the tgz:

getstarted-nodejs> npm-bundle

Finally, deploy the new version of your application to AWS:

getstarted-nodejs> boxfuse run -env=prod

Fusing Image for getstarted-nodejs-0.0.1.tgz ...
Image fused in 00:02.679s (15077 K) -> myuser/getstarted-nodejs:0.0.1
Pushing myuser/getstarted-nodejs:0.0.1 ...
Verifying myuser/getstarted-nodejs:0.0.1 ...
Waiting for AWS to create an AMI for myuser/getstarted-nodejs:0.0.1 in eu-central-1 (this may take up to 50 seconds) ...
AMI created in 00:28.465s in eu-central-1 -> ami-738b6c1c
Creating security group boxsg-myuser-prod-getstarted-nodejs-0.0.1 ...
Launching t2.micro instance of myuser/getstarted-nodejs:0.0.1 (ami-738b6c1c) in prod (eu-central-1) ...
Instance launched in 00:42.912s -> i-3649978b
Waiting for AWS to boot Instance i-3649978b and Payload to start at https://52.59.244.133/ ...
Payload started in 00:11.316s -> https://52.59.244.133/
Remapping Elastic IP 52.58.12.244 to i-3649978b ...
Waiting 15s for AWS to complete Elastic IP Zero Downtime transition ...
Terminating instance i-af499712 ...
Destroying Security Group sg-1defe174 (boxsg-myuser-prod-getstarted-nodejs-0.0.0) ...
Deployment completed successfully. myuser/getstarted-nodejs:0.0.1 is up and running at https://getstartednodejs-myuser.boxfuse.io/

And there it is:

Summary

In this post, we saw how to deploy and update a Node.js application to AWS using CloudCaptain in 3 easy steps.

First we fused our application into a 15 MB CloudCaptain image in under 3 seconds and ran an instance of it on VirtualBox. We then deployed the same image unchanged to AWS. And finally we updated our application on AWS with zero downtime using blue/green deployments.

To do so we used CloudCaptain and its 3 core principles: Immutable Infrastructure, Minimal Images and Blue/Green deployments.

And that's not it! CloudCaptain also supports dead-easy debugging, auto-scaling, database auto-provisioning and much more.

So if you haven't already, sign up for your CloudCaptain account now (simply log in with your GitHub id, it's free) to start deploying your Node.js applications effortlessly on AWS in minutes.

Debugging and Profiling

2016-02-18T00:00:00+00:00

CloudCaptain makes it trivial to fuse and run your JVM-based applications unchanged both on VirtualBox and on AWS. This perfect environment parity, where your machine in prod is absolutely identical to the one in dev, makes it much easier to discover and correct issues sooner. But sometimes you still need to dig deeper into your running application to fully understand what's going on.

Today we are introducing support for dead easy debugging and profiling, both locally on VirtualBox and in the cloud on AWS.

Debugging

When the logs aren't helping anymore, it is time to roll up your sleeves and pull up your favorite debugger to step through the code and truly understand exactly what's going on. CloudCaptain now makes it trivial to remotely debug your apps, regardless of whether they run on VirtualBox or on AWS.

Start by launching an instance set up for remote debugging:

> boxfuse run myapp.jar -debug

...

Ready to accept remote debugger connections -> tcp://127.0.0.1:5005

The -debug flag opens an additional port to which your debugger can connect. As the JVM debugging protocol doesn't have any authentication, by default this port is restricted to only allow connections from your local machine.

Now pull up your favorite IDE and connect its debugger to the address shown above:

And that's it! Simply add breakpoints, step through the code, and evaluate expressions as you normally would, regardless of whether your instance runs on VirtualBox or on AWS.

Profiling

Do you suspect a memory leak, a thread deadlock or unusually high CPU usage? CloudCaptain now makes it trivial to profile your apps, regardless of whether they run locally or in the cloud.

Start by launching an instance set up for JMX remote management:

> boxfuse run myapp.jar -jvm.jmx

...

Ready to accept JMX connections -> 127.0.0.1:1099 (user: boxfuse / password: boxfuse)

The -jvm.jmx flag opens an additional port to which your JMX client (like Java VisualVM or Java Mission Control) can connect. By default this port is restricted to only allow connections from your local machine.

Profiling with Java VisualVM

Simply launch Java VisualVM:

> jvisualvm

And connect to the address above:

And that's it! Go ahead and profile and analyse CPU, memory and threads as you normally would:

Profiling and Managing with Java Mission Control

Alternatively you can also use the newer Java Mission Control to profile and manage your app:

> jmc

Simply connect Java Mission Control to the address above by creating a JVM connection:

Finally select MBean Server and profile CPU, memory and threads as well as inspect and invoke actions on MBeans as you normally would:

As you would expect this of course works identically on both VirtualBox and AWS.

Summary

In this post, we introduced dead easy debugging and profiling for CloudCaptain instances, both locally on VirtualBox and in the cloud on AWS.

CloudCaptain now makes it trivial to remotely debug your application directly from your favorite IDE. Additionally you can now enable JMX remote management to easily profile and manage your JVMs using tools like Java VisualVM and Java Mission Control.

So if you haven't already, simply log in to the CloudCaptain Console with your GitHub id and download the CloudCaptain Client to run, debug and profile your JVM application on AWS today. Enjoy!

Hello MySQL!

2016-02-12T00:00:00+00:00

In our previous post we introduced CloudCaptain's new support for managing databases. In addition to being able to quickly fuse VM images and run those unchanged locally on VirtualBox and in the Cloud on AWS, CloudCaptain now also makes it trivial to run the relational databases that those services need.

Today we are extending CloudCaptain's database support beyond PostgreSQL and adding full support for MySQL.

Additionally we are spicing up our database management with two new great features: boxfuse info and boxfuse open -db.

MySQL

No more management headaches for your MySQL databases. CloudCaptain now takes care of provisioning, starting, stopping and deleting them for you (CloudCaptain always creates a snapshot before deleting a database, so you can restore it at any time).

What do you need to do to enable this? You can either specify that you want a MySQL database when you create your app via your client of choice (Command-line tool, Maven plugin or Gradle plugin) or simply the CloudCaptain Console:

However in most cases you won't need to even do this. If your application includes the MySQL JDBC driver CloudCaptain will detect it automatically for you and configure your application accordingly. If want to manage your database yourself, this can of course be disabled by explicitly setting the database type to none when you create your app.

CloudCaptain then automatically provisions a MySQL database for you. It does this both locally using the CloudCaptain Dev VM and on AWS using RDS.

And just like for PostgreSQL, if your application is based on Spring Boot, Grails, Play or Dropwizard, CloudCaptain will automatically configure your framework's default datasource with the correct database driver class name, URL, user and password. This works identically in every environment, both locally and on AWS.

boxfuse info

Sometimes however you also want to access your database from outside the application itself. To make it easy to get the database url, user name, password and more we are introducing a new command called boxfuse info. This command gives you all the information you need:

> boxfuse info hello -env=test

Info about myuser/hello in the test environment:

App Type    : Single Instance with Zero Downtime updates
App URL     : https://hello-test-myuser.boxfuse.io/
DB Type     : PostgreSQL database
DB URL      : jdbc:postgresql://boxdb-myuser-test-hello.ca53c5vrhrzn.eu-central-1.rds.amazonaws.com:5432/hello
DB Host     : boxdb-myuser-test-hello.ca53c5vrhrzn.eu-central-1.rds.amazonaws.com
DB Port     : 5432
DB Database : hello
DB User     : qbGh5bYandLAYiXBHXp5L8jQliU9qwAkEFRf1R7JzJJ3azNwWmauKfYJe1TCoUW
DB Password : JDBsCkFuDUCqLwDhbWZcZWmaeFCN7W8aAIzy0MjYA5ICXKCf3zMO7azw9092VFFqsnKayqWefAaklZchfATAbLsvShdsOWjKJBaU
DB Status   : available

You can then easily copy this information to configure the client of your choice.

boxfuse open -db

While being to copy this information is nice, we think not having to copy-paste anything is even better!

To be able to quickly access the database (MySQL or PostgreSQL) of your app in any environment using that database's native client (mysql or psql) via a secure connection with the character encoring properly set, we are introducing a new option to the open command: boxfuse open -db.

So now if you run an application on your test environment on AWS and you want to access its database (say PostgreSQL in this case) running on RDS securily via SSL, all you need to do is:

> boxfuse open hello -env=test -db

Launching psql to access the myuser/hello PostgreSQL database in the test environment ...
psql (9.4.5, Server 9.4.5)
SSL connection (Cipher: ECDHE-RSA-AES256-SHA, Bits: 256)
Type "help" for help.

hello=>\conninfo
You are connected with the database „hello“ as user „qbGh5bYandLAYiXBHXp5L8jQliU9qwAkEFRf1R7JzJJ3azNwWmauKfYJe1TCoUW“ on host „boxdb-myuser-test-hello.ca53c5vrhrzn.eu-central-1.rds.amazonaws.com“ on port „5432“.
SSL connection (Cipher: ECDHE-RSA-AES256-SHA, Bits: 256)
hello=>\q

And there you have it! You are now is a secure connection with your database using its native client. All the character encoding and RDS root certificate management is done automatically for you.

Summary

In this post, we introduced full MySQL support for CloudCaptain's dead-easy database management capabilities.

CloudCaptain now automatically detects the MySQL JDBC driver in your application and proceeds to provision MySQL databases both locally and on AWS RDS. CloudCaptain exposes the database URL, user and password as environment variables within the application instances. And if your application is based on Spring Boot, Grails, Play or Dropwizard, CloudCaptain automatically configures your framework's default datasource with those settings.

However sometimes you also want to access your database through some external tool, so we introduced boxfuse info to be able to quickly and conveniently get the information you need.

But if you want an even easier solution you can now use CloudCaptain's new boxfuse open -db switch to drop straight into a secure connection to the database of your application in that environment using your database's native client.

So if you haven't already, start deploying your applications and their databases effortlessly with CloudCaptain today. Simply log in to the CloudCaptain Console with your GitHub id, download the CloudCaptain Client and enjoy!

Introducing Database support

2016-01-28T00:00:00+00:00

In 2015 CloudCaptain made it dead easy to run & auto-scale JVM applications on AWS. We added special optimizations for Dropwizard, Spring Boot, Play and Grails in addition to the support for executable jars, Tomcat and TomEE (for Java EE).

While this provides a great help for tackling the complexity of the compute tier, many of our users who build microservices that follow the database per service pattern have been asking for built-in support for relational databases.

Today we are introducing dead-easy PostgreSQL database management for all environments, both locally using a new dev VM and on AWS using RDS.

What's included?

So what exactly is included in this feature? CloudCaptain is now able to provision PostgreSQL databases both locally and on AWS.

On your local machine, CloudCaptain will launch a new dev VM alongside with your application that contains a fully configured PostgreSQL database. Your applications instances will then automatically receive environment variables containing the URL, user and password required to connect to that database. And if your application is based on Spring Boot, Grails, Play or Dropwizard CloudCaptain will automatically configure your framework to use that database for its default datasource.

On AWS, CloudCaptain will automatically provision either single- or multi-AZ RDS instances for your application. You can easily scale those to different instance types or with additional storage. Just like on your local machine, your applications instances will automatically receive environment variables containing the URL, user and password required to connect to that database. And again if your application is based on Spring Boot, Grails, Play or Dropwizard CloudCaptain will automatically configure your framework to use that database for its default datasource.

Let's explore this

To give you a better impression of this new feature, we'll walk you through a simple example where we build a Spring Boot microservice that uses a geo-replicated highly available PostgreSQL database.

Prerequisites

Before we get started ensure you have created a CloudCaptain account (it's free, just log in with your GitHub id). Also make sure to associate it with your AWS account in the CloudCaptain console to be able to deploy on EC2.

You will also need a JDK and VirtualBox installed on your machine.

Step 0: Creating the Spring Boot microservice

Let's first create a basic Spring Boot application. Go to start.spring.io, select the PostgreSQL, JDBC and Web dependencies and click Generate Project:

Now unpack your project in a new directory and delete the src/test (we won't need it in this tutorial). You should have the following structure:

 database-demo
   src
     main
       java
         boxfusedemo
           DatabaseDemoApplication.java
       resources
         static
         templates
         application.properties
     mvnw
     mvnw.cmd
     pom.xml

Change the project version in pom.xml to 1.0:

<version>1.0</version>

Next, add Flyway as a dependency to pom.xml so we can create the database schema we'll need:

<dependency>
    <groupId>org.flywaydb</groupId>
    <artifactId>flyway-core</artifactId>
</dependency>

And create a migration as src/main/resources/db/migration/V1__Messages.sql:

CREATE TABLE messages (
  id INT NOT NULL PRIMARY KEY,
  msg TEXT NOT NULL
);

INSERT INTO messages (id, msg) VALUES (1, 'Greetings from Spring Boot auto-configured by CloudCaptain!');

Then add a simple controller to boxfusedemo.DatabaseDemoApplication:

package boxfusedemo;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import javax.sql.DataSource;

@SpringBootApplication
@RestController
public class DatabaseDemoApplication {
    @Autowired
    private DataSource dataSource;

    @RequestMapping("/")
    public String hello() {
        return new JdbcTemplate(dataSource).queryForObject("select msg from messages where id=?", String.class, 1);
    }

    public static void main(String[] args) {
        SpringApplication.run(DatabaseDemoApplication.class, args);
    }
}

And last but not least, compile it into an executable jar file:

> mvnw package -DskipTests

Great. Your Spring Boot microservice executable jar is now available under target/database-demo-1.0.jar.

Step 1: Fusing a CloudCaptain image and running it on VirtualBox

Now it's time to fuse your application into a CloudCaptain image and launch an instance of it on VirtualBox:

> boxfuse run

Fusing Image for database-demo-1.0.jar ...
Image fused in 00:07.373s (56827 K) -> myuser/database-demo:1.0
Downloading boxfuse-dev-hdd 20160123 ...
Preparing CloudCaptain Dev VM HDD ...
Creating CloudCaptain Dev VM for myuser/database-demo ...
Exposing Dev VM PostgreSQL port on localhost:5432
Starting CloudCaptain Dev VM for myuser/database-demo ...
Launching Instance of myuser/database-demo:1.0 on VirtualBox ...
Forwarding http port localhost:8080 -> vb-db91551d:8080
Instance launched in 00:03.426s -> vb-db91551d
Waiting for Payload to start on Instance vb-db91551d ...
Payload started in 00:13.157s -> https://127.0.0.1:8080

What just happened here?

CloudCaptain detected the Maven project structure and automatically found the executable jar. CloudCaptain then determined that this is a Spring Boot application that includes the PostgreSQL driver and automatically configured it to require a PostgreSQL database. It then fused our application's executable into a minimal bootable VM image.

Before starting an instance of our image, CloudCaptain installed and launched a dev VM that contains a fully-configured PostgreSQL database. The PostgreSQL service from that VM was then exposed to our app on port 5432.

CloudCaptain then launched a VM instance of our application's image and automatically injected the correct database URL, user and password as environment variables. Those environment variables were then picked up by our Spring Boot application to instantiate and autowire the datasource required by our small rest controller.

Seeing it in action

To see our controller in action, simply point your browser to https://127.0.0.1:8080 or type:

> boxfuse open

And here our service successfully returning the record from our freshly created and migrated database:

Step 2: Deploying to AWS

Now let's deploy our microservice to AWS. As CloudCaptain works with your AWS account, it first needs the necessary permissions to do so. So if you haven't already done it, go to the CloudCaptain Console and connect your AWS account now.

Every new CloudCaptain account comes with 3 environments: dev, test and prod. dev is your local VirtualBox environment and test and prod are on AWS.

So let's deploy our microservice to the prod environment on AWS:

> boxfuse run -env=prod

Creating myuser/database-demo ...
Pushing myuser/database-demo:1.0 ...
Verifying myuser/database-demo:1.0 ...
Creating security group boxsg-db-myuser-prod-database-demo ...
Creating RDS postgresql DB (db.t2.micro / 5 GB) => boxdb-myuser-prod-database-demo (this one-time action may take up to 10 minutes to complete) ...
Waiting for AWS to create an AMI for myuser/database-demo:1.0 in eu-central-1 (this may take up to 50 seconds) ...
AMI created in 00:32.075s in eu-central-1 -> ami-9d1d05f1
Waiting for AWS to make RDS DB boxdb-myuser-prod-database-demo available ...
DB boxdb-myuser-prod-database-demo [creating]
DB boxdb-myuser-prod-database-demo [backing-up]
DB boxdb-myuser-prod-database-demo [available]
Creating Elastic IP ...
Mapping databasedemo-myuser.boxfuse.io to 52.29.173.190 ...
Creating security group boxsg-myuser-prod-database-demo-1.0 ...
Launching t2.micro instance of myuser/database-demo:1.0 (ami-9d1d05f1) in prod (eu-central-1) ...
Instance launched in 00:49.422s -> i-d671f06a
Waiting for AWS to boot Instance i-d671f06a and Payload to start at https://54.93.73.84:8080/ ...
Payload started in 00:55.276s -> https://54.93.73.84:8080/
Remapping Elastic IP 52.29.173.190 to i-d671f06a ...
Waiting 15s for AWS to complete Elastic IP Zero Downtime transition ...
Deployment completed successfully. myuser/database-demo:1.0 is up and running at https://databasedemo-myuser.boxfuse.io:8080/

What just happened here?

CloudCaptain pushed our image to the secure CloudCaptain Vault. CloudCaptain then automatically provisioned an RDS PostgreSQL database (and its associated security group). It then went on to create an AMI for our image. Once our database was ready, CloudCaptain then create a new domain name and mapped it to a new elastic ip. It then created a security group four our app and launch an instance of our AMI.

CloudCaptain automatically injected the correct database URL, user and password as environment variables. Those environment variables were then picked up by our Spring Boot application to once again instantiate and autowire the datasource required by our small rest controller.

Seeing it in action

Once again, to see our controller in action, simply type:

> boxfuse open -env=prod

And here our service successfully returning the record from our freshly created and migrated database:

Step 3: Making our database highly available

Finally let's make sure our infrastructure can survive in the face of failure. To achieve that we're going to make our database highly available by provisioning a continuously replicated geo-redundant hot standby spare instance that's ready take over in case the primary one fails.

To achieve this we'll simply scale our database in the CloudCaptain Console:

And that's it! Not only is our database now able to automatically fail-over, but we now also get near zero-downtime (typically only a few seconds) when patching or minor version upgrades are required.

Summary

In this post, we introduced CloudCaptain's new dead-easy database management capabilities.

CloudCaptain is now able to automatically provision PostgreSQL databases both locally and on AWS RDS. CloudCaptain exposes the database URL, user and password as environment variables within the application instances. And if your application is based on Spring Boot, Grails, Play or Dropwizard, CloudCaptain will automatically configure your frameworks's default datasource with those settings.

Finally making your database highly available and scaling both the instance type and the allocated storage is now a breeze. All it takes is a few clicks in the CloudCaptain Console.

So if you haven't already, sign up for your CloudCaptain account now (simply log in with your GitHub id, it's free) and start deploying your applications and their PostgreSQL databases effortlessly on AWS today.

Deploy Grails Apps effortlessly to AWS with Gradle

2015-12-23T00:00:00+00:00

Groovy, Grails and Gradle together form an excellent and highly productive platform for developing modern web applications.

Today we're going to look at deploying Grails applications written in Groovy effortlessly to AWS using Gradle and CloudCaptain.

We'll do so using the 3 core principles behind CloudCaptain:

1.	Immutable Infrastructure	Creating servers and never modifying them again by treating a server as one immutable unit that is regenerated after every change and promoted unchanged from environment to environment. This eliminates drift and increases reliability by ensuring you run the exact same code in production as the code you tested in test.
2.	Minimal Images	Analysing your application and generating minimal tailor-made Linux-based images on the fly that are 100x smaller than a typical Linux system and take just seconds to produce.
3.	Blue/Green deployments	Deploying a new version of an app in parallel to the existing one and only making the switch at the elastic IP or elastic load balancer level once the configured health checks of the new version have passed. Deployments are fully automated and effectively transactional, providing you with zero-downtime updates.

The plan

Our plan is to first create a new Grails app which we'll fuse into a minimal CloudCaptain image that can be deployed unchanged both on VirtualBox and AWS.

We'll then deploy our image to VirtualBox (great for rapid feedback and quick local tests), push it to the CloudCaptain Vault (our secure online repository) and run it on AWS.

And finally we'll update it on AWS with zero-downtime blue/green deployments.

Prerequisites

You will also need a JDK, Grails and VirtualBox installed on your machine.

Step 0: Creating a Grails application

Start by creating a Grails application:

> grails create-app getstarted-grails

Navigate to the newly created directory:

> cd getstarted-grails

And build an executable jar file:

getstarted-grails> grails war

Great. Your Grails application executable jar is now available under build/libs/getstarted-grails-0.1.jar.

Step 1: Fusing a CloudCaptain image and running it on VirtualBox

Before we can begin we first have to add the CloudCaptain Gradle plugin to our build.gradle:

plugins {
    id "com.boxfuse.client" version "1.37.0.2024"
    ...
}

boxfuse {
    // You can find these in the Downloads tab of the CloudCaptain Console (https://console.cloudcaptain.sh/#/downloads)
    user='your-boxfuse-client-user'
    secret='your-boxfuse-client-secret'
}

Now it's time to fuse your application into a CloudCaptain image and launch an instance of it on VirtualBox:

getstarted-grails> gradle boxfuseRun

This command analyses your application and generates a minimal Linux-based image for it. It does this by combining your application itself with a JRE and a Linux kernel from the CloudCaptain Component Inventory. It will then launch an instance of your new image on VirtualBox. This entire process should take around 10 seconds.

When it completes you simply open a browser pointing to your fresh new VirtualBox VM:

getstarted-grails> gradle boxfuseOpen

And here it is:

The port this instance is running on wasn't picked by chance. CloudCaptain analyses your native application.yml Grails config file and automatically extracts the correct port and healthcheck configuration.

You can also see this information in your list of images:

getstarted-grails> gradle boxfuseLs

Images available locally:
+------------------------------+---------------------------+-------+---------+-----------+--------------+---------+---------------------+
| Image                        |          Payload          | Debug |  Java   | AppServer |    Ports     |  Size   |    Generated at     |
+------------------------------+---------------------------+-------+---------+-----------+--------------+---------+---------------------+
| myuser/getstarted-grails:0.1 | getstarted-grails-0.1.jar | false | 8.60.22 | Grails    | http -> 8080 | 97828 K | 2015-12-21 17:47:50 |
+------------------------------+---------------------------+-------+---------+-----------+--------------+---------+---------------------+
Total: 1

As well as in the list of running instances:

getstarted-grails> gradle boxfusePs

Running Instances on VirtualBox in the dev environment :
+-------------+------------------------------+---------------------+-----------------------+---------------------+
|  Instance   |            Image             |        Type         |          URL          |     Launched at     |
+-------------+------------------------------+---------------------+-----------------------+---------------------+
| vb-9feb5d7c | myuser/getstarted-grails:0.1 | 4 CPU / 1024 MB RAM | https://127.0.0.1:8080 | 2015-12-21 17:47:56 |
+-------------+------------------------------+---------------------+-----------------------+---------------------+
Total: 1

Step 2: Deploying your application to AWS

Now let's deploy your image to AWS. As CloudCaptain works with your AWS account, it first needs the necessary permissions to do so. So if you haven't already done it, go to the CloudCaptain Console and connect your AWS account now.

Every new CloudCaptain account comes with 3 environments: dev, test and prod. dev is your local VirtualBox environment and test and prod are on AWS.

So let's deploy our application to the prod environment on AWS:

getstarted-grails> gradle boxfuseRun -Dboxfuse.env=prod -i

...
Pushing myuser/getstarted-grails:0.1 ...
Verifying myuser/getstarted-grails:0.1 ...
Waiting for AWS to create an AMI for myuser/getstarted-grails:0.1 in eu-central-1 (this may take up to 50 seconds) ...
AMI created in 00:19.095s in eu-central-1 -> ami-fd5b4491
Creating Elastic IP ...
Mapping getstartedgrails-myuser.boxfuse.io to 52.29.77.147 ...
Creating security group boxsg-myuser-prod-getstarted-grails-0.1 ...
Launching t2.micro instance of myuser/getstarted-grails:0.1 (ami-fd5b4491) in prod (eu-central-1) ...
Instance launched in 00:50.707s -> i-2c1daf90
Waiting for AWS to boot Instance i-2c1daf90 and Payload to start at https://52.59.247.126:8080/health ...
Payload started in 00:55.567s -> https://52.59.247.126:8080/health
Remapping Elastic IP 52.29.77.147 to i-2c1daf90 ...
Waiting 15s for AWS to complete Elastic IP Zero Downtime transition ...
Deployment completed successfully. myuser/getstarted-grails:0.1 is up and running at https://getstartedgrails-myuser.boxfuse.io:8080/

With that one command CloudCaptain has automatically pushed your existing image to the CloudCaptain Vault, the CloudCaptain secure only image repository, as well as provisioned, configured and secured all necessary AWS resources including a domain name, an elastic IP, a security group and your instance itself. There is no manual work necessary on your behalf.

Once again CloudCaptain automatically used the port and healthcheck information extracted from your application.yml Grails configuration file.

All you need to do now is simply navigate to your new domain to see your Grails application in action on AWS:

Step 3: Updating your application using blue/green deployments

Now let's take things one step further and update your application on AWS with zero downtime using blue/green deployments.

This fits perfectly with our model of immutable infrastructure where we never update an instance in-place, and instead launch a new one to replace it. The new instance is only put into service once all healthchecks have passed and it has proven to be fit for the job. This ensures to you always have a working application up and running serving your users.

So first let's update our app by modifying grails-app/views/index.gsp and change the title:

<h1>Welcome to Grails on CloudCaptain!</h1>

then bump the version in build.gradle:

version "0.2"

and rebuild the jar and fuse a new CloudCaptain image:

getstarted-grails> grails clean
getstarted-grails> gradle boxfuseFuse -Dboxfuse.env=prod

Finally deploy the new version of your application to AWS:

getstarted-grails> gradle boxfuseRun -Dboxfuse.env=prod -i

...
Pushing myuser/getstarted-grails:0.2 ...
Verifying myuser/getstarted-grails:0.2 ...
Waiting for AWS to create an AMI for myuser/getstarted-grails:0.2 in eu-central-1 (this may take up to 50 seconds) ...
AMI created in 00:18.013s in eu-central-1 -> ami-7b5a4517
Creating security group boxsg-myuser-prod-getstarted-grails-0.2 ...
Launching t2.micro instance of myuser/getstarted-grails:0.2 (ami-7b5a4517) in prod (eu-central-1) ...
Instance launched in 00:41.174s -> i-b502b009
Waiting for AWS to boot Instance i-b502b009 and Payload to start at https://52.59.254.138:8080/health ...
Payload started in 01:27.645s -> https://52.59.254.138:8080/health
Remapping Elastic IP 52.29.77.147 to i-b502b009 ...
Waiting 15s for AWS to complete Elastic IP Zero Downtime transition ...
Terminating instance i-2c1daf90 ...
Destroying Security Group sg-5981cb30 ...
Deployment completed successfully. myuser/getstarted-grails:0.2 is up and running at https://getstartedgrails-myuser.boxfuse.io:8080/

And there it is:

Summary

In this post, we saw how to deploy and update a Grails application to AWS using Gradle and CloudCaptain in 3 easy steps. First we fused our application into a CloudCaptain minimal image and ran an instance of it on VirtualBox. We then deployed the same image unchanged to AWS. And finally we updated our application on AWS with zero downtime using blue/green deployments.

To do so we used CloudCaptain and its 3 core principles: Immutable Infrastructure, Minimal Images and Blue/Green deployments.

So if you haven't already, sign up for your CloudCaptain account now (simply log in with your GitHub id, it's free) and start deploying and auto-scaling your Groovy and Grails applications effortlessly on AWS today.

Expanding fast AMI creation to us-west-2

2015-12-17T00:00:00+00:00

When we first launched CloudCaptain we immediately introduced support for all AWS regions, but only one of them was optimized for AMI creation speed (eu-central-1).

As CloudCaptain usage grew, we expanded our fast AMI creation capabilities to us-east-1 and eu-west-1 to meet customer demand.

So why is it important to have a region optimized for speed?

We'll let the numbers speak for themselves:

As you can see even though even the slowest CloudCaptain regions are already faster than Packer, moving to a CloudCaptain optimised region gives you a major boost, bringing AMI creation to about 30 seconds, which is an order of magnitude faster than most other tools in the industry.

Expanding fast AMI creation to us-west-2

Starting today, you can now also enjoy the same ultra-fast AMI creation performance in the us-west-2 region. No need to change anything on your end. Simply fire up your CloudCaptain Client and start enjoying the ultra-fast performance now:

> boxfuse run hello -env=prod

Creating myuser/hello ...
Pushing myuser/hello:1.0 ...
Verifying myuser/hello:1.0 ...
Waiting for AWS to create an AMI for myuser/hello:1.0 in us-west-2 (this may take up to 50 seconds) ...
AMI created in 00:32.692s in us-west-2 -> ami-fd213e9c
Creating Elastic IP ...
Mapping hello-myuser.boxfuse.io to 52.33.26.21 ...
Creating security group boxsg-myuser-hello-1.0 ...
Launching t2.micro instance of myuser/hello:1.0 (ami-fd213e9c) in prod (us-west-2) ...
Instance launched in 00:27.148s -> i-413c4698
Waiting for AWS to boot Instance i-413c4698 and Payload to start at https://54.201.148.195/ ...
Payload started in 00:33.618s -> https://54.201.148.195/
Remapping Elastic IP 52.33.26.21 to i-413c4698 ...
Waiting 15s for AWS to complete Elastic IP Zero Downtime transition ...
Deployment completed successfully. myuser/hello:1.0 is up and running at https://hello-myuser.boxfuse.io/

As CloudCaptain expands its footprint, further regions will be optimized for fast deployment. The order will be based on based on customer usage and demand. This does not affect the runtime speed of your instance, which is determined by the instance type you selected.

Conclusion

We have taken CloudCaptain's ultra-fast AMI creation capabilities in the eu-central-1, us-east-1 and eu-west-1 regions and expanded them to the us-west-2 region. No change is necessary on your end. Simply use CloudCaptain to create AMIs and deploy applications as you always do and automatically start enjoying the absolute fastest way to create AMIs today.

Have fun and keep on deploying JVM applications to AWS with ease and pleasure!

And if you haven't already, sign up for your CloudCaptain account now. All you need is a GitHub user and you'll be up and running in no time. The CloudCaptain free plan aligns perfectly with the AWS free tier, so you can deploy your JVM application to EC2 completely free.

Provisioning temp space

2015-12-10T00:00:00+00:00

CloudCaptain makes it effortless to deploy modern JVM-based applications on VirtualBox and AWS. Most of these applications are based on Spring Boot, Dropwizard, Tomcat, TomEE or the Play Framework and many of them are completely stateless with all state stored in some external service like RDS or S3.

This works great as it perfectly fits the core principles behind CloudCaptain: Immutable Infrastructure, Minimal Images and Blue/Green deployments.

However some applications also need to temporarily store some state locally for a number of reasons including:

locally caching large objects
storing intermediate steps of a larger job
staging file uploads

Introducing easy temp space provisioning

Starting today CloudCaptain can now easily provision any amount of temp space you need. Every instance going forward will have 1 GB of temp space on disk. You can increase this all the way to 16384 GB (16 TB) or disable it completely by setting it to 0. This storage should be considered strictly ephemeral and will be automatically destroyed once the instance it is attached to terminates.

So how does this work in practice?

Say you have an application that requires 100 GB of temp space. You can now enable that by doing:

> boxfuse run myapp-1.0.jar -tmp=100

And that's it. CloudCaptain will fuse an image and launch an instance by provisioning both the root device (where your application code resides) and a new temp volume of 100 GB which will be automatically available for you to use under /tmp. CloudCaptain will also automatically configure the JVM to use the /tmp folder for all temporary files.

You are fully flexible and can enable this either at image creation or at instance launch time.

Also note that on AWS all temp volumes are encrypted by default and auto-destroy when the instance terminates to ensure the privacy of your data.

Conclusion

While CloudCaptain has always worked great with modern stateless JVM applications, today we introduced support for easily provisioning from 0 to 16384 GB (16 TB) of strictly ephemeral instance temp space. This space is automatically ready to use and mounted under /tmp.

This feature is available today for all CloudCaptain users on both VirtualBox and AWS instances at no additional cost. (AWS charges may apply)

Existing users only need to update their CloudCaptain client with

> boxfuse -u

to start using this.

Enjoy!

And if you haven't already, sign up for your CloudCaptain account now (simply log in with your GitHub id, it's free) and start deploying and auto-scaling your JVM applications effortlessly on AWS today.

Deploy Play Framework Scala Apps effortlessly to AWS

2015-11-16T00:00:00+00:00

When it comes to creating modern JVM-based applications we have a number of excellent choices including Spring Boot, Dropwizard, Tomcat, TomEE and the Play Framework.

Today we're going to look at deploying Play applications written in Scala effortlessly to AWS using CloudCaptain.

We'll do so using the 3 core principles behind CloudCaptain:

1.	Immutable Infrastructure	Creating servers and never modifying them again by treating a server as one immutable unit that is regenerated after every change and promoted unchanged from environment to environment to eliminate drift and increase reliability by ensuring you run the exact same code in production as the one you tested in test.
2.	Minimal Images	Analysing your application and generating minimal tailor-made Linux-based images on the fly that are 100x smaller than a typical Linux system and take just seconds to produce.
3.	Blue/Green deployments	Deploying a new version of an app in parallel to the existing one and only making the switch at the elastic IP or elastic load balancer level once the configured health checks of the new version have passed. Deployments are fully automated and effectively transactional, providing you with zero-downtime updates.

Prerequisites

Before we get started ensure you have created a CloudCaptain account (it's free, just log in with your GitHub id). Make sure to associate it with your AWS account in the CloudCaptain console to deploy on EC2.

You will also need a JDK, the Typesafe Activator and VirtualBox installed on your machine.

Step 0: Creating the Play application

Start by creating a Play Scala application using the Typesafe Activator:

> activator new hello play-scala

And navigate to the newly created directory:

> cd hello

Then set the version in build.sbt to 1.0:

version := "1.0"

Finally build a distribution zip:

hello> activator dist

Great. Your Play application distribution zip is now available under target/universal/hello-1.0.zip.

Step 1: Fusing a CloudCaptain image and running it on VirtualBox

Now it's time to fuse your application into a CloudCaptain image and launch an instance of it on VirtualBox:

hello> boxfuse run

This command will analyse your application and generate a minimal Linux-based image for it, which we call a Bootable App. It does this by combining your application itself with a JRE and a Linux kernel from the CloudCaptain Component Inventory. It will then launch an instance of your new image on VirtualBox. When it completes you should see a message like this:

Payload started in 00:06.688s -> https://127.0.0.1:10080

Now open your browser and navigate to this address to see your new application up and running within the new VirtualBox VM:

You can also see your newly created image:

hello> boxfuse ls

Images available locally:
+------------------+---------------+-------+---------+-----------+------------+---------+---------------------+
| Image            |    Payload    | Debug |  Java   | AppServer |   Ports    |  Size   |    Generated at     |
+------------------+---------------+-------+---------+-----------+------------+---------+---------------------+
| myuser/hello:1.0 | hello-1.0.jar | false | 8.60.22 | Play      | http -> 80 | 75950 K | 2015-11-12 14:57:50 |
+------------------+---------------+-------+---------+-----------+------------+---------+---------------------+
Total: 1

As well as the instance that is running:

hello> boxfuse ps

Running Instances on VirtualBox in the dev environment :
+-------------+------------------+---------------------+------------------------+---------------------+
|  Instance   |      Image       |        Type         |           URL          |     Launched at     |
+-------------+------------------+---------------------+------------------------+---------------------+
| vb-b18d6746 | myuser/hello:1.0 | 4 CPU / 1024 MB RAM | https://127.0.0.1:10080 | 2015-11-12 14:57:56 |
+-------------+------------------+---------------------+------------------------+---------------------+
Total: 1

Step 2: Deploying your application to AWS

Every new CloudCaptain account comes with 3 environments: dev, test and prod. dev is your local VirtualBox environment and test and prod are on AWS.

So let's deploy our application to the prod environment on AWS:

hello> boxfuse run -env=prod

Pushing myuser/hello:1.0 ...
Verifying myuser/hello:1.0 ...
Waiting for AWS to create an AMI for myuser/hello:1.0 in eu-central-1 (this may take up to 50 seconds) ...
AMI created in 00:34.394s in eu-central-1 -> ami-ed988b81
Creating Elastic IP ...
Mapping hello-myuser.boxfuse.io to 52.28.107.167 ...
Creating security group boxsg-myuser-prod-hello-1.0 ...
Launching t2.micro instance of myuser/hello:1.0 (ami-ed988b81) in prod (eu-central-1) ...
Instance launched in 00:26.922s -> i-5de042e1
Waiting for AWS to boot Instance i-5de042e1 and Payload to start at https://52.28.121.55/ ...
Payload started in 00:51.516s -> https://52.28.121.55/
Remapping Elastic IP 52.28.107.167 to i-5de042e1 ...
Waiting 15s for AWS to complete Elastic IP Zero Downtime transition ...
Deployment completed successfully. myuser/hello:1.0 is up and running at https://hello-myuser.boxfuse.io/

Notice that CloudCaptain has reused your image unchanged instead fusing a new one.

With that one command CloudCaptain has automatically pushed your image to the CloudCaptain Vault, our secure only image repository, as well as provisioned, configured and secured all necessary AWS resources including a domain name, an elastic IP, a security group and your instance itself. There is no manual work necessary on your behalf.

All you need to do is simply navigate to your new domain to see your Play application in action on AWS:

For our Play Framework backend, we used CloudCaptain to minimize the effort required to deploy updates and hotfixes to AWS. Now we can focus fully on improving the application itself. Further, the support is fantastic - quick, friendly and helpful. More than one would expect for such a low price.

Konrad Peters, Lead Developer, Appguru

Step 3: Updating your application using blue/green deployments

Now let's take things one step further and deploy an update of your application with zero downtime using blue/green deployments.

Start by modifying app/controllers/Application.scala with a simple change:

package controllers

import play.api._
import play.api.mvc._

class Application extends Controller {
  def index = Action {
    Ok(views.html.index("Updated by CloudCaptain with zero downtime!"))
  }
}

then bump the version in build.sbt:

version := "1.1"

and rebuild the dist:

hello> activator clean dist

Finally, deploy the new version of your application to AWS:

hello> boxfuse run -env=prod

Fusing Image for hello-1.1.zip ...
Image fused in 00:09.817s (75949 K) -> myuser/hello:1.1
Pushing myuser/hello:1.1 ...
Verifying myuser/hello:1.1 ...
Waiting for AWS to create an AMI for myuser/hello:1.1 in eu-central-1 (this may take up to 50 seconds) ...
AMI created in 00:34.152s in eu-central-1 -> ami-8b988be7
Creating security group boxsg-myuser-prod-hello-1.1 ...
Launching t2.micro instance of myuser/hello:1.1 (ami-8b988be7) in prod (eu-central-1) ...
Instance launched in 00:35.372s -> i-ebea4857
Waiting for AWS to boot Instance i-ebea4857 and Payload to start at https://52.29.129.239/ ...
Payload started in 01:05.316s -> https://52.29.129.239/
Remapping Elastic IP 52.28.107.167 to i-ebea4857 ...
Waiting 15s for AWS to complete Elastic IP Zero Downtime transition ...
Terminating instance i-5de042e1 ...
Destroying Security Group sg-48ea9921 ...
Deployment completed successfully. myuser/hello:1.1 is up and running at https://hello-myuser.boxfuse.io/

And there it is:

Summary

In this post, we saw how to deploy and update a Play Framework Scala application to AWS in 3 easy steps. First we fused our application into an minimal image called a Bootable App and ran an instance of it on VirtualBox. We then deployed the same image unchanged to AWS. And finally we updated our application on AWS with zero downtime using blue/green deployments.

To do so we used CloudCaptain and its 3 core principles: Immutable Infrastructure, Minimal Images, Blue/Green deployments.

So if you haven't already, sign up for your CloudCaptain account now (simply log in with your GitHub id, it's free) and start deploying and auto-scaling your Play applications effortlessly on AWS today.

It's Auto-Scaling time!

2015-10-26T00:00:00+00:00

Two of the key design principles of CloudCaptain are reliable and fast generation of minimal immutable images and outstanding integration integration with AWS.

Today we're going to leverage both together to take things even further.

AWS' native format for images is the AMI, the Amazon Machine Image. It is a central building block of EC2. Every single EC2 instance you launch is based on an AMI. And CloudCaptain makes it really easy to create them.

Immutable images such as AMIs also have the great advantage that they really simplify scaling. Whether you have to set up 1, 10, 100 or 1000 instances, all you need to do is launch that number of identical copies from the same base image and you're done. Refreshingly simple.

In fact CloudCaptain has enabled you to do this for a while. Regardless of whether you want to scale horizontally, vertically or both, CloudCaptain simply lets you do:

> boxfuse scale myapp -env=prod -capacity=7:t2.large

This will effectively launch 7 new t2.large EC2 instances of your app, and once the healthchecks have passed, CloudCaptain will terminate any previous deployments and perform a zero-downtime transition.

While this is great for guaranteeing you always have the required amount of processing power available, it forces you to guess your capacity needs in advance, which means that at certain times you may be paying for more capacity than you actually need.

Auto-Scaling

Today we are introducing CloudCaptain support for AWS Auto-Scaling. You can now let AWS automatically optimize the scale of your system based on the current load.

All you need to do is drag a slider to indicate the minimum and the maximum number of instances you want, and specify at which CPU usage thresholds the scaling activities should happen, and click Scale. CloudCaptain and AWS will take care of the rest.

In this example here, CloudCaptain will automatically configure AWS CloudWatch to monitor the CPU usage of your instances. CloudCaptain will also automatically configure Auto-Scaling to kick in as soon as the rolling average over a period of 300 seconds exceeds 75%. Auto-Scaling will then start provisioning additional EC2 instances up to a maximum of 10. If CPU usage instead drops to 25% or below, EC2 instances will be terminated down to a minimum of 2.

The same can of course also be specified from the command-line:

> boxfuse scale myapp -env=prod -capacity=2-10:t2.micro:cpu25-75:300

Successfully configured myapp to auto-scale between 2 and 10 t2.micro instances based on average CPU load over 300 seconds, scaling in at 25% and below, scaling out at 75% and above.

And voila! Done! Auto-scaling has never been this easy!.

Tip: To best optimize your budget with your actual usage, start with auto-scaling the smallest type of general purpose instances like t2.micro and only move to larger or more specialized instances if needed.

Summary

Immutable Infrastructure makes scaling fast and reliable. However having fixed capacity for your system requires you to guess your needs in advance. It also means that at times you'll potentially pay more than you should.

Starting today, CloudCaptain can now automatically configure AWS to auto-scale your application based on average CPU usage. AWS will add or remove capacity from the system fully automatically based on the current load, so that you never have to worry again about paying for more than you actually need.

This is available today at no extra cost to all paid plans.

Have fun and keep on deploying JVM applications to AWS with ease and pleasure!

Introducing Restricted Ports

2015-10-20T00:00:00+00:00

At its core, CloudCaptain analyses the payload you pass it (like a .jar or .war file) and generates a minimal immutable image for it. This image contains two main parts: a disk image and metadata.

The metadata is what interests us here. It is built as part of the analysis phase and contains information about your app, such as the ports and the healthcheck urls it requires. If you use Spring Boot or Dropwizard this info can even be automatically extracted for you from these frameworks' native configuration files.

CloudCaptain then later uses this metadata when running instances of your images to correctly provision infrastructure. In the case of AWS, CloudCaptain will use the port and healthcheck info to automatically set up and configure Elastic Load Balancers, Security Groups and more.

However so far, unless you explicitly configured a custom security group, any port you have opened was always open to the world:

While this is great for publicly facing parts of the service like http(s) ports it may not always be what you want for other parts of it like the admin interface for example.

Restricted Ports

Today we are introducing a new CloudCaptain feature called Restricted Ports. What this allows you, is to further constrain the access rights of a specific port down to a specific range of ip addresses.

We have added support for fixed addresses like 12.34.56.78 as well as CIDR blocks such as 12.34.56.00/24 (granting access to all ip addresses between 12.34.56.00 and 12.34.56.255).

Additionally we have also added support for a special address denoted by @, which refers to the current public ip of your local workstation. This is particularly useful when working with protocols which lack authentication such as JVMTI for debugging. You can now open a debug port that is only accessible from your own IP address.

If your ISP assigns you dynamic addresses you can combine these approaches and grant access to the CIDR block of your current IP such as @/22.

This is how this then works out in practice if you want to for example restrict the access of the admin port of your app to the current ip of your workstation:

UDP Ports

Another feature we added is support for UDP as not every service needs the additional features (and overhead) of TCP. To achieve this you can simply add /udp to the port definition:

> boxfuse run -ports.myudpsrvc=6543/udp

This will then effectively expose UDP port 6543 to the world. This is of course combinable with the restricted ports described above.

Summary

While certain protocols or interfaces aren't meant to be publicly available, yet are an important part of your application, we have introduced Restricted Ports to make it easy to constrain access to a single address or a CIDR block. We have also added support for the convenience value @ to easily refer to the current public ip address of your workstation. On top of that it is now possible to also open UDP ports.

Here is for reference the complete list of currently supported syntax variants for port definition:

`80`	TCP port 80, universally accessible
`80/tcp`	TCP port 80, universally accessible
`80/udp`	UDP port 80, universally accessible
`80/tcp:@`	TCP port 80 only accessible from your own IP
`80/udp:@/20`	UDP port 80 only accessible from the IPs in the CIDR /20 block of your own IP
`80/udp:1.2.3.4`	UDP port 80 only accessible from 1.2.3.4
`80/tcp:1.2.3.4/31`	TCP port 80 only accessible from the IPs in the CIDR /31 block of 1.2.3.4

Have fun and keep on deploying JVM applications to AWS with ease and pleasure!

Expanding fast AMI creation to us-east-1 and eu-west-1

2015-10-13T00:00:00+00:00

The main cornerstone to making the transition to Immutable Infrastructure a success is having the capability to produce machine images quickly and reliably These images can then be deployed across environments and ensure perfect environment parity from dev to prod.

CloudCaptain has from the get-go been able to produce minimal machine images tailor-made for your applications on the OS of your choice. No matter whether you choose to use Windows, Mac or Linux, CloudCaptain always produces the same consistent and reliable images in under 10 seconds.

You can then run VirtualBox instances based on those images locally to iterate rapidly.

Once you are satisfied with the results, you can then push these images to the CloudCaptain Vault and convert them to AWS AMIs to be able to launch instances on EC2.

CloudCaptain has always been about an order of magnitude faster than other competing products when performing this conversion in the eu-central-1 region:

Expanding fast AMI creation to us-east-1 and eu-west-1

Starting today, you can now also enjoy the same ultra-fast AMI creation performance in the us-east-1 and eu-west-1 regions. No need to change anything on your end. Simply fire up your CloudCaptain Client and start enjoying the ultra-fast performance now:

> boxfuse run hello -env=prod

Creating axelfontaine/hello ...
Pushing axelfontaine/hello:1.0 ...
Verifying axelfontaine/hello:1.0 ...
Waiting for AWS to create an AMI for axelfontaine/hello:1.0 in us-east-1 (this may take up to 50 seconds) ...
AMI created in 00:45.467s in us-east-1 -> ami-c1743ea4
Creating Elastic IP ...
Mapping hello-axelfontaine.boxfuse.io to 52.22.39.178 ...
Creating security group boxsg-axelfontaine/hello:1.0 ...
Launching t2.micro instance of axelfontaine/hello:1.0 (ami-c1743ea4) in prod (us-east-1) ...
Instance launched in 00:48.109s -> i-ec71ce53
Waiting for AWS to boot Instance i-ec71ce53 and Payload to start at https://54.165.227.247/ ...
Payload started in 00:43.000s -> https://54.165.227.247/
Remapping Elastic IP 52.22.39.178 to i-ec71ce53 ...
Waiting 15s for AWS to complete Elastic IP Zero Downtime transition ...
Deployment completed successfully. axelfontaine/hello:1.0 is up and running at https://hello-axelfontaine.boxfuse.io

These are the current AMI creation times across all regions:

ap-northeast-1	Less than 5 minutes
ap-southeast-1	Less than 5 minutes
ap-southeast-2	Less than 5 minutes
eu-central-1	Less than 60 seconds
eu-west-1	Less than 60 seconds
sa-east-1	Less than 5 minutes
us-east-1	Less than 60 seconds
us-west-1	Less than 5 minutes
us-west-2	Less than 5 minutes

Conclusion

We have taken CloudCaptain's ultra-fast AMI creation capabilities in the eu-central-1 region and expanded them to the us-east-1 and eu-west-1 regions. No change is necessary on your end. Simply use CloudCaptain to create AMIs and deploy applications as you always do and automatically start enjoying the absolute fastest way to create AMIs today.

Have fun and keep on deploying JVM applications to AWS with ease and pleasure!

Custom Environments and Multi-Region Apps

2015-10-06T00:00:00+00:00

At CloudCaptain we are strong believers in Immutable Infrastructure. You generate an image once and you then deploy it across environments. All instances are based on the exact same set of bits and bytes. There is no SSH to avoid drift and to ensure perfect environment parity from dev to prod.

By default every new account comes with three environments:

dev	Develop with fast turnarounds on your local machine using VirtualBox
test	Test your applications on AWS in a production-like environment
prod	Run your applications in production on AWS

Introducing Custom Environments

While dev, test and prod are sufficient for most projects, a number of you have come to us and asked for the ability to define additional environments like staging for example. Today we are please to announce that this is now the case. All users on paid plans can start creating custom environments in the CloudCaptain Console with a few clicks:

The CloudCaptain Client works with them as expected, simply specify the new environment and you're good to go:

> boxfuse run myapp -env=staging

Introducing Multi-Region Apps

While custom environments are great, we decided to take things further. You can now create custom environments in AWS regions that differ from your default CloudCaptain AWS region by simply selecting the target region while creating the custom environment:

This effectively lets you easily create multi-region apps.

Conclusion

While dev, test and prod are sufficient for most projects, CloudCaptain now has you covered when you grow beyond that. All users on paid plans can now define custom environments and start creating multi-region apps with ease and pleasure.

Enjoy and read more in the documentation.

Have fun! And if you haven't already, sign up for your CloudCaptain account now. All you need is a GitHub user and you'll be up and running in no time. The CloudCaptain free plan aligns perfectly with the AWS free tier, so you can deploy your Java application to EC2 completely free.

Hello Gradle Plugin Portal!

2015-09-30T00:00:00+00:00

Build tool integration has always been a priority for us at CloudCaptain. It is an essential part of a smooth development and deployment workflow. Back in May we extended our support beyond Maven and released to the CloudCaptain Gradle plugin, both of which have been steadily improved ever since.

Gradle Plugin Portal

Recently Gradle has been making it easier and easier to both find and use plugins. There is now an official Gradle Plugin portal, making it easy to discover new plugins and keep track of new releases.

We are happy to announce that as of today the CloudCaptain Gradle Plugin is now available there too:

And that's is just the beginning. If you are using Gradle 2.1 or newer you can now also use the new and improved plugin syntax:

plugins {
    id "com.boxfuse.client" version "1.37.0.2024"
}

Using CloudCaptain with Gradle has never been easier!

Deploy Java EE Apps effortlessly on AWS

2015-09-28T00:00:00+00:00

At CloudCaptain we love solutions that just work. And that's why, when it comes to Java EE, we've grown very fond of Apache TomEE. Unlike traditional heavy and complex application servers, TomEE combines the swiftness of Tomcat with the benefits of standard Java EE.

Today we are happy to announce that we have added TomEE to CloudCaptain's selection of payloads that can be deployed effortlessly to VirtualBox and AWS. Just like Dropwizard, Spring Boot, Tomcat and (executable) jar files, you can now fuse TomEE applications into immutable images with a single command. These images can then be deployed unchanged to VirtualBox and AWS with zero downtime updates.

Why TomEE?

At its core Apache TomEE stands for 3 things:

Be small
Be Java EE certified
Be Tomcat

TomEE does away with long and tedious deployments typically associated with Java EE. Instead you now get a Tomcat that has been tuned to be able to run Java EE applications. It's still Tomcat, but it is now also fully Java EE certified with support for EJB, CDI, JMS and more. It starts quickly and does away with .ear files. Instead you can simply use tried and true .war files and define any JNDI resources in WEB-INF/resources.xml. Java EE has never been this simple!

Prerequisites

You will also need Git, Java, Maven and VirtualBox installed on your machine.

Creating a TomEE Java EE app

Start by cloning the CloudCaptain Samples repository:

> git clone https://github.com/boxfuse/boxfuse-samples.git

and build the TomEE sample application:

> cd tomee
tomee> mvn package

Great. You should now have a file called target/tomee-1.0.war as part of the following application:

This application uses various Java EE features including EJBs, JSPs, Servlets, JPA and JNDI.

Deploying it to AWS EC2

Now that you have built the application, let's deploy it to AWS. CloudCaptain will start by analysing it and determining its type. Doing so it will find the WEB-INF/resources.xml file. This tells it that it is a Java EE that requires TomEE instead of Tomcat.

CloudCaptain will then create a Micro-OS on-the-fly designed to do one thing and one thing only: run this version of your application.

It does that by combining your application with TomEE, a JVM, the C library, the Linux kernel and a bootloader into an tiny image. Everything is fully configured and ready to run out of the box:

CloudCaptain Image -> AWS Instance">

To launch your application on AWS as well as provision and configure all necessary infrastructure, all you need to do is:

> boxfuse run target/tomee-1.0.war -env=prod

Fusing Image for tomee-1.0.war ...
Image fused in 00:08.592s (73858 K) -> axelfontaine/tomee:1.0
Creating axelfontaine/tomee ...
Pushing axelfontaine/tomee:1.0 ...
Verifying axelfontaine/tomee:1.0 ...
Waiting for AWS to create an AMI for axelfontaine/tomee:1.0 in eu-central-1 (this may take up to 50 seconds) ...
AMI created in 00:18.531s -> ami-806e6c9d
Creating Elastic IP ...
Mapping tomee-axelfontaine.boxfuse.io to 52.29.32.202 ...
Creating security group boxsg-prod-axelfontaine/tomee:1.0 ...
Launching t2.micro instance of axelfontaine/tomee:1.0 (ami-806e6c9d) in prod (eu-central-1) ...
Instance launched in 00:38.394s -> i-dec80062
Waiting for AWS to boot Instance i-dec80062 and Payload to start at https://52.29.28.195/ ...
Payload started in 01:00.050s -> https://52.29.28.195/
Remapping Elastic IP 52.29.32.202 to i-dec80062 ...
Waiting 15s for AWS to complete Elastic IP Zero Downtime transition ...
Deployment completed successfully. axelfontaine/tomee:1.0 is up and running at https://tomee-axelfontaine.boxfuse.io

And there you have it, your Java EE application is now fully up and running on AWS:

You can deploy new versions at any time and CloudCaptain will automatically perform blue/green zero downtime updates for you.

Summary

In this post you saw how Apache TomEE makes Java EE fast and lightweight. We took this to the next level by using CloudCaptain to deploy a Java EE application to AWS using a single command. There was no tedious configuration and the entire process took less than two minutes, including automatically provisioning all the necessary infrastructure on EC2.

Running your Java EE applications on AWS has never been this easy.

Immutable Infrastructure: Fast & Reliable Scaling

2015-08-26T00:00:00+00:00

Moving to Immutable Infrastructure brings many advantages. In the previous article in this series we saw how it eliminates drift and the problems associated with deferred provisioning by capturing your application and all its dependencies at the lowest level possible: an image of the entire system. We also looked at how to enforce immutability by creating images with no SSH. With this technique any change requires rebuilding a new image through your regular automated build and deployment pipeline. In turn your system drastically gains in simplicity and reliability.

Another area where Immutable Infrastructure radically simplifies things is scaling. Let's find out how.

Why scaling?

Scaling is adding or removing capacity from your system. The main driver for scaling has traditionally been load. When load increases beyond what can be reliably handled by the current infrastructure, more capacity must be provisioned to handle it.

However today we have another important force at play: money. On the public cloud, cost is an important driver for the way we architect our systems. In stark contrast to on premise setups where hardware is almost universally a sunk cost, public clouds like AWS offer utility-like pricing. You essentially pay for what you use. It therefore becomes essential to not only be able to quickly add, but also to remove capacity from a system! You effectively optimize your budget to optimally handle the current load on the system.

Horizontal vs Vertical Scaling

When it comes to scaling, it really boils down to two options. Changing the size or the number of instances. You can either scale horizontally by adding (scaling out) or removing (scaling in) instances, or you can scale vertically to bigger (scaling up) or smaller (scaling down) machines.

Regardless of the scaling direction and technique you choose, you need to be able to do this quickly and reliably. While this sounds easy, its has traditionally come with a number of challenges, the most common ones being state, provisioning times and robustness. However as you'll see these aren't intrinsic problems to scaling. They come from the way we have traditionally handled state and provisioned machines.

The challenges with traditional provisioning

As we already saw in the previous article traditional provisioning with shell scripts or configuration management tools is very prone to drift as it flies in the face of the proven mantra: build your artifact only once.

The reason for this, is that instead of building an image with all software fully configured, secured and ready to boot, this provisioning process happens on every instance you launch. This puts you at risk for all issues related to deferred provisioning as well as all potential reliability problems of having to fetch everything again and again from remote repositories which may have changed or temporarily gone offline.

The other big issue with this is that it takes a very long time. Whereas with Immutable Infrastructure and fully-provisioned images all you need to do is make a copy of the volume and start up the instance, with traditional provisioning you first have to go through a tedious installation process before your application can even begin to start up.

Dealing with state

When it comes to dealing with state, Immutable Infrastructure forces you to do the right thing: keep all persistent state off the instance. This doesn't mean you can't have a local copy as a a cache to speed things up (beware of stale data though!), but the master should be in a central highly-available and replicated data store.

This applies first and foremost to application data which should reside in data stores, in-memory caches or object storage. But also to other things like log files which in 2015 should not be spread across multiple instances, but instead aggregated in a central log server where they can be archived, indexed, and made searchable. And last, but not least, also sessions should be moved to either a cookie, a data store or a central in-memory cache. We'll talk more in detail about this in a future post.

Conclusion

In a public cloud world you need to be able to quickly and reliably add capacity to a system in response to load, but also remove capacity to optimize your budget. You can do this by scaling horizontally (in & out) and adjust the number of instances. Or you can do it by scaling vertically (up & down) to change the size of your instances.

Traditional provisioning poses a number of challenges related to drift, deferred provisioning, reliability and performance. Immutable Infrastructure solves these problems through fast and reliable launching of instances based on fully provisioned images. It also forces you to do the right thing and keep all persistent state off instance.

Stay tuned for future blog blog posts where we'll explore more aspects of Immutable Infrastructure including sessions and service discovery.

Try it for yourself

CloudCaptain lets you do all this and more. CloudCaptain intelligently analyses your application and generates minimal images in seconds. There is no general purpose operating system and no tedious provisioning. CloudCaptain images are lean, secure and efficient. You can run them on VirtualBox for development and deploy them unchanged and with zero downtime on AWS for test and production.

CloudCaptain Convert: fully-provisioned AMIs in 30 seconds

2015-08-19T00:00:00+00:00

One of the key elements of Immutable Infrastructure is the capability to efficiently create fully-provisioned images. These images must have all required software already on board. In fact, with everything fully configured and ready to run on boot, these images don't even need SSH.

On AWS, the typical process to achieve that can be described as follows:

You first create an AMI, based on which you can then launch any number of identical instances.

Integrating with Existing Workflows

CloudCaptain lets accomplish all this a single command, boxfuse run, that will effectively fuse your application artifact into an image, convert CloudCaptain image to an AWS AMI and launch the number of instances you desire based on the newly created AMI.

Some of you however have already been practicing this for a long time with various combinations of open-source and in-house tools. You have asked us to provide you with a gradual migration process to adopt CloudCaptain one piece at a time.

And that's why today we are introducing boxfuse convert to allow you to benefit from CloudCaptain's incredibly fast AMI creation, while still relying on your current instance provisioning technology:

With boxfuse convert you can finally say goodbye to your old, slow and flaky AMI creation workflows. This is all you need to do to go from a JVM-based application (using Dropwizard in this example) to a fully-provisioned, secure, production-ready AMI:

> boxfuse convert dwunikernel-1.0.jar

Fusing Image for dwunikernel-1.0.jar ...
Image fused in 00:06.525s (55362 K) -> axelfontaine/dwunikernel:1.0
Pushing axelfontaine/dwunikernel:1.0 ...
Verifying axelfontaine/dwunikernel:1.0 ...
Waiting for AWS to create an AMI for axelfontaine/dwunikernel:1.0 in eu-central-1 (this may take up to 50 seconds) ...
AMI created in 00:19.777s -> ami-e66264fb

And there you have it:

Your brand new fully-provisioned AMI has been created from scratch in less than 30 seconds! It is tagged consistently with boxfuse:app and boxfuse:image tags so you can retrieve it quickly and reliably. You can immediately start launching instances based upon it from the AWS console or your orchestration tool of choice.

Comparing to Packer

Packer is a great tool for building images for many platforms. It is not without its problems though. Besides the fact that Packer effectively builds an image per platform (instead of using the same image on all platforms), it is also incredibly slow when it comes to building AMIs. For comparison, try building the same type of AMI (HVM+EBS in this case) using Packer and Ubuntu 14.04 LTS. And don't forget to go and grab a coffee, as it will take a while. Compared to Packer's EBS backed AMI builder, CloudCaptain is more than 10 times faster:

And that's not all. CloudCaptain's AMIs are also more than 10 times smaller:

(In fact CloudCaptain AMIs are even much smaller than that, but AWS's smallest EBS snapshot size is a full 1 GB)

And finally consider the effort required to write and maintain your Packer .json descriptors (don't forget to pin the version of every single package to avoid the issues of deferred provisioning) to create fully-provisioned hardened images that are ready to boot without manual tuning and with no SSH.

Now look at CloudCaptain Convert's one-liner again and see for yourself just how easy things can be.

How does CloudCaptain do it?

CloudCaptain intelligently analyses your application and generates minimal images in seconds. There is no general purpose operating system and no tedious provisioning. CloudCaptain images are lean, secure and efficient. You can run them on VirtualBox for development and deploy them unchanged and with zero downtime on AWS for test and production.

So if you haven't already, sign up for your CloudCaptain account now. All you need is a GitHub account and you'll be up and running in no time. The CloudCaptain free plan gives you 100 AMI builds per month. And if you wish to take things further it also aligns perfectly with the AWS free tier, so you can deploy your application to EC2 completely free.

Immutable Infrastructure Talk at DevNexus 2015

2015-08-11T00:00:00+00:00

This year has seen plenty of CloudCaptain events and we have plenty more coming up!

Today we are happy to share with you the recording of the highly acclaimed DevNexus 2015 talk about Immutable Infrastructure and CloudCaptain by Axel Fontaine, founder and CEO of CloudCaptain.

What attendees had to say about it

@axelfontaine was phenomenal presenting the mind blowing idea of "Immutable Infrastructure" - #devnexus
— Ali Faisal (@alifais) March 11, 2015

At a really fun presentation by @axelfontaine on immutable app/server architecture #devnexus
— Kevin (@anjunatl) March 11, 2015

So if you weren't there, regardless of whether you are new to Immutable Infrastructure and CloudCaptain or if you are already on board and what to check out some best practices, follow this attendee's advice:

Y'all are missing @axelfontaine's session "Immutable Infrastructure: the new App Deployment"... Catch the recording later! #devnexus
— Adam DeJardine (@adamdejardine) March 11, 2015

Video recording of the talk

Slides

Try it now

Have fun! And if you haven't already, sign up for your CloudCaptain account now. CloudCaptain intelligently analyses your application and generates minimal images in seconds. There is no general purpose operating system and no tedious provisioning. CloudCaptain images are lean, secure and efficient. You can run them on VirtualBox for development and deploy them unchanged and with zero downtime on AWS for test and production. All you need is a GitHub user and you'll be up and running in no time. The CloudCaptain free plan aligns perfectly with the AWS free tier, so you can deploy your application to EC2 completely free.

CloudCaptain Events in Q3 2015

2015-07-21T00:00:00+00:00

Always wanted to learn more about Immutable Infrastructure and the No SSH movement? Want to see CloudCaptain in action and talk to like-minded forward thinking people? Look no further. We have a number of exciting events lined up in Q3 across North America and Europe.

Intro to CloudCaptain

12th August 2015 in West Chester, Pennsylvania, USA

Learn to ship your products with Shippable and CloudCaptain. Join the DevOps 2.0 revolution with continuous delivery and manage deployments of multi-tier applications without code, scripts, or IT automation tools.

Presented by Andrew Schwabe, Founder and CEO of Formatic.ly

Spring Boot mit AWS und CloudCaptain

2nd September 2015 in Kassel, Germany

Presented by Waldemar Schneider, Senior Developer at Flavia

Up in the sky: Architecture and Deployment of Microservices for the Cloud

10th September 2015 in Oslo, Norway

The case for running an own datacenter is vanishing rapidly. The cloud seduces with a low barrier of entry and full flexibility. Infrastructure can be spun up almost instantly in a fully automated way through an API. All this with no upfront costs and the ability to decommission it just as fast. But what does this mean for our applications and their architecture? Can we just lift and shift them to the cloud? Everything comes at a price. To be able to fully leverage the potential of the cloud, new challenges must be mastered: from data privacy and security to cost-based architectures, dynamic provisioning, service discovery and efficient deployment models. This talk provides architects and developers clear answers and battle-tested solutions for a successful journey to infrastructure heaven.

Presented by Axel Fontaine, Founder and CEO of CloudCaptain

If you want to learn more, we also put together a brand new page with all past and future CloudCaptain events.

Immutable Infrastructure: No SSH

2015-07-13T00:00:00+00:00

One of the things that is really exciting about Immutable Infrastructure is that it opens up a vast number of opportunities to revisit old ways and change them for the better. One of those is drift, the slow natural divergence of machines from each other and their intended setup. There are two main causes for this: deferred provisioning and updates. Both are exacerbated by time. The longer apart machines are set up and the longer they exists, the higher the likely hood to run into drift problems. Let's look at each of these in turn.

Why is drift a problem?

The first question to ask is really: why is drift itself a problem? Or to put it differently: why is having identical machines important?

One of the primary ways to reduce risk in a software system is testing. Both manual and automated testing rely on the same three step workflow:

Put the system in a known state
Perform an action
Compare the results against your expectations

Putting the system in a known state does not only apply to data, it also applies to the versions of all the software components installed. Once your system is correctly set up, your tests will then validate version X of your code running on version Y of your platform while having version Z of a library on board. All other combinations are unknown and must be validated separately. In other words, there is no guarantee that the exact version of your code will work identically when combined with older or newer versions of the platform and libraries as older versions may still contain bugs and newer versions could have regressions or subtle changes.

For the investment you have made with your tests to have any value, you must therefore ensure the known state of the system and the versions of all components can be easily and safely replicated across machines and environments.

Or to put it more simply: to have any guarantee that things will work, you need to run in production what you tested in test.

The same thing of course also applies within a single environment. If you have multiple machines serving clients behind a load balancer and want to keep your sanity, you also want these to be absolutely identical.

The problems with deferred provisioning

Now you may think this is easily solved by using some solid automated provisioning mechanism. Regardless of whether you favor shell scripts or configuration management tools, this is not as easy as it seems. The problem lies in the common usage patterns of package managers and update timings in central repositories. Trouble starts the minute you start relying on commands like this:

> sudo apt-get install mypkg

To better illustrate why this is a problem, look at this simple example of three machines being set up based on the exact same provisioning script at different points in time:

As you can see, even though all machines install Package A and Package B, the difference in provisioning time causes the same script to pull different versions of the same packages onto the various machines. You effectively couple the stability of your system to the update schedule of individual packages in the central repository!

Images to the rescue

One way to fix this is to pin all packages and all their transitive dependencies to specific versions. While this lets your regain control of the update schedule of your system, this comes with a high maintenance overhead.

A simpler and more reliable alternative is to stick to the proven mantra: build your artifact only once.

To eliminate differences (and thus potential causes of failure) between environment, your build should produce an artifact that captures your application and all its dependencies at the lowest level possible. When working with virtual infrastructure, the lowest level possible is simply an image of the entire machine. From then on identical instances can be created when you want as often as you want:

While images make it easy to create identical machines, this doesn't solve all problems. As soon as you have the possibility to log in you lose guarantee that those machines will not diverge over time:

The longer those machines those machines exists, the more likely someone will have modified something and the harder they become to recreate exactly. This effectively largely removes a lot of the benefits.

Enforcing immutability

So what can be done to remedy this? You need to prevent log in. This not only removes the drift problem. It also has very compelling security implications by reducing your attack surface. Vulnerabilities like ShellShock simply vanish.* And on top of that it forces you to stick to a clean image-based workflow, without hacks or back-doors. In light of all these advantages we'd like to invite you to join us on the no SSH movement:

Your infrastructure now effectively becomes immutable. Any change requires rebuilding of a new image through your regular automated build and deployment pipeline. In turn your system drastically gains in simplicity and reliability. A pretty good deal if you ask us.

Conclusion

Drift reduces the reliability of your system by effectively undermining the guarantees you gained while testing it. Yet it very easily occurs when relying on traditional provisioning.

By moving to a build and deployment pipeline that captures your application and all of its dependencies at the lowest possible level, you can regain a lot of confidence. When dealing with virtual hardware, the best way to achieve this is to produce images of an entire system. To ensure the process stays reliable and to keep you honest you should eliminate SSH and the ability to log in. Your images and your infrastructure effectively becomes immutable. Any change triggers the creation of a new image, which is built only once for all environments.

* Update: As correctly pointed out by a few people, the ShellShock vulnerability doesn't fully vanish with this in a general-purpose operating system. This is a CloudCaptain-specific point as CloudCaptain doesn't include bash either.

Take it to 11

Deploying Spring Boot Apps on EC2

2015-06-10T00:00:00+00:00

Lately the JVM-world has finally started embracing solutions that just work. Gone are the days where you had to manually wire together many components and write large amounts of XML configuration just to make things work. Today it is all about being productive from the get go. And to us this is really music to our ears. Our products (Flyway & CloudCaptain) have always adhered to this exact design principle from the start. And so regardless of whether you easily want to migrate your relational databases or fuse your applications into bootable apps, Flyway & CloudCaptain work very had to make you successful. Rapidly and reliably.

Pivotal has now gone the same way and tied their entire portfolio of Spring frameworks and libraries into a fantastic framework called Spring Boot. It is a turn-key solution to the Spring ecosystem.

At its core, Spring Boot combines your application with an embedded container and all its other libraries into a single executable jar that can easily be deployed onto any system. It adds lots of auto-configuration and integration options to make your application production-ready from day one. In fact Spring Boot even offers out of the box integration for Flyway so adopting industry best practices like database migrations has never been easier.

In this post we'll take Spring Boot's idea of executable jars to its natural conclusion: fusing your Spring Boot application together with a JVM and a kernel into a immutable bootable app that can be run unchanged on both VirtualBox and EC2.

Prerequisites

You will also need Java, Maven and VirtualBox installed on your machine.

Creating a Spring Boot app

Creating a new Spring Boot app is incredibly easy. The Spring Boot website provides a simple service at start.spring.io to create application skeletons directly via http. To create the application for this post, you can either click to generate the application and extract it or type this one-liner:

> curl "https://start.spring.io/starter.zip?baseDir=springbootec2&groupId=boxfusedemo&artifactId=springbootec2&version=1.0&name=springbootec2&description=Demo+project+for+Spring+Boot+on+EC2+blog+post+at+https%3A%2F%2Fcloudcaptain.sh%2Fblog%2Fspring-boot-ec2.html&packageName=springbootec2&type=maven-project&packaging=jar&javaVersion=1.8&language=java&bootVersion=1.2.4.RELEASE&style=web&style=actuator&generate-project=" > springbootec2.zip && unzip springbootec2.zip

Now change to the newly created directory and you'll find the following structure:

To be able to interact with the application you'll need a controller, so go ahead and create one as springbootec2.HelloController with the following contents:

package springbootec2;
    
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

import java.util.HashMap;
import java.util.Map;

@RestController
public class HelloController {
    @RequestMapping(value = "/hello", method = RequestMethod.GET)
    public Map<String, Object> hello(@RequestParam(value = "name", defaultValue = "CloudCaptain") String name) {
        Map<String, Object> result = new HashMap<>();
        result.put("greeting", "Hello " + name + "!");
        return result;
    }
}

Your application is now fully set up. As your skeleton also includes Spring Boot's actuator (their turn-key production readiness features), you now have two urls available:

/hello : your controller
/health : Spring Boot's health check page

Go ahead and try it by executing

> mvn package

followed by

> java -jar target/springbootec2-1.0.jar

You should now be able to see it in action at https://localhost:8080/hello?name=world:

Fusing it into a CloudCaptain Bootable App

To ensure an application runs reliably on another environment, always strive to package and transport it at the lowest level possible. Spring Boot already helps you part of the way by embedding the application server and all other other libraries directly into the executable jar. This is great as it avoids the need to separately install and update an application on each individual instance. However you are still responsible for ensuring the base operating system, its libraries and the JVM are properly installed, updated and configured.

Spring Boot -> CloudCaptain">

CloudCaptain starts where Spring Boot leaves off. It takes your Spring Boot executable jar and fuses it with a JVM and a Linux kernel into a single immutable unit called a Bootable App. This Bootable App is created in seconds and takes just 1% of the size of a regular system. It only contains your app and its direct dependencies. Nothing else. CloudCaptain does away with the operating system and the provisioning as you know it. Instead it analyses your application and generates a very efficient tailor-made image on the fly. This image can then be launched unchanged on VirtualBox and AWS. No intermediate OS, runtime daemon or other installation is required. It runs directly on your virtualization layer.

To fuse your Spring Boot executable jar into a CloudCaptain Bootble simply issue this command:

> boxfuse fuse target/springbootec2-1.0.jar

Fusing Image for springbootec2-1.0.jar ...
Image fused in 00:06.423s (53253 K) -> axelfontaine/springbootec2:1.0

In just a few seconds CloudCaptain took your Spring Boot executable jar (12 MB) and fused it into a 53 MB image containing a Java 8 JVM and a kernel:

> boxfuse ls

Images available locally:
+--------------------------------+-----------------------+-------+---------+-------------+------------+---------+
| Image                          |        Payload        | Debug |  Java   |  AppServer  |   Ports    |  Size   |
+--------------------------------+-----------------------+-------+---------+-------------+------------+---------+
| axelfontaine/springbootec2:1.0 | springbootec2-1.0.jar | false | 8.45.14 | Spring Boot | http -> 80 | 53253 K |
+--------------------------------+-----------------------+-------+---------+-------------+------------+---------+
Total: 1

CloudCaptain has built-in deep integration for Spring Boot applications. It automatically detects the Spring Boot actuator health check urls as well as the port configuration.

To ensure your application can configure itself correctly, CloudCaptain automatically activates a Spring profile called boxfuse on startup.

Time to see it in action. Launch your shiny new bootable app on VirtualBox with one command:

> boxfuse run springbootec2:1.0

Launching Instance of axelfontaine/springbootec2:1.0 on VirtualBox ...
Forwarding http port localhost:8888 -> vb-9b5be89b:80
Instance launched in 00:06.005s -> vb-9b5be89b
Waiting for Payload to start on Instance vb-9b5be89b ...
vb-9b5be89b => instance boot messages omitted
vb-9b5be89b => ...
vb-9b5be89b => ...
Payload started in 00:07.490s -> https://127.0.0.1:8888

Your VirtualBox instance is now up and running as you can see in your browser:

Running it unchanged on EC2

Now that you've successfully fused your Spring Boot executable jar into an image and booted it on VirtualBox, it is time to launch an instance of it on EC2. By default CloudCaptain supports 3 environments: dev (default, VirtualBox), test (AWS) and prod (AWS).

So in this case, all you need to do is switch the environment to prod:

> boxfuse run springbootec2:1.0 -env=prod

Creating axelfontaine/springbootec2 ...
Pushing axelfontaine/springbootec2:1.0 ...
Verifying axelfontaine/springbootec2:1.0 ...
Creating Elastic IP ...
Mapping springbootec2-axelfontaine.boxfuse.io to 52.28.128.122 ...
Waiting for AWS to create an AMI for axelfontaine/springbootec2:1.0 in eu-central-1 ...
AMI created in 00:31.553s -> ami-f88ab3e5
Creating security group boxfuse-sg_axelfontaine/springbootec2:1.0 ...
Launching t2.micro instance of axelfontaine/springbootec2:1.0 (ami-f88ab3e5) in eu-central-1 ...
Instance launched in 00:21.765s -> i-0ae46fcb
Waiting for Payload to start on Instance i-0ae46fcb at https://52.28.132.192/health ...
Remapping Elastic IP 52.28.128.122 to i-0ae46fcb ...
Waiting 15s for AWS to complete Elastic IP Zero Downtime transition ...
Deployment completed successfully. axelfontaine/springbootec2:1.0 is up and running at https://springbootec2-axelfontaine.boxfuse.io

And there you have it:

So what just happened here? Quite a few things! In about 60 seconds CloudCaptain

encrypted and signed your image (springbootec2:1.0)
uploaded it to the CloudCaptain Vault & verified the signature
created a new AWS elastic IP (52.28.128.122)
created a new domain name (springbootec2-axelfontaine.boxfuse.io)
mapped the domain name to the elastic ip
converted your image to an AMI (Amazon Machine Image)
created an AWS security group based on the port information it detected when fusing the image
launched a t2.micro instance based on the newly created AMI
checked the health of your app at /health based on the Spring Boot actuactor configuration
remapped the AWS elastic IP to your new instance

All of this with a single command and no additional configuration required.

You can also integrate CloudCaptain directly into your build by using either the CloudCaptain Maven Plugin or the CloudCaptain Gradle Plugin.

Summary

In this post you saw why Spring Boot is a great way to be productive from the get go with Spring. We saw that by embedding the application server and all other libraries directly into an executable jar, you gain greater consistency between environments. We then took this idea to its natural conclusion and used CloudCaptain to fuse the Spring Boot executable in a Bootable App that bundles all direct dependencies of the application all the way down to the OS kernel. The image was created in seconds and measured just a few MBs. We ran it on VirtualBox and deployed it unchanged in under 60 seconds on AWS with a single command. What you have here is the simplest and most reliable way to run your Spring Boot applications on EC2.

You can read more about CloudCaptain's first class support for Spring Boot as well as how to integrate it directly into your build with the CloudCaptain Maven & Gradle plugins. Find out more about AWS support, deploying Elastic Load Balancers and multiple instances and more in the documentation.

Hello Gradle!

2015-05-26T00:00:00+00:00

Here are CloudCaptain we pride ourselves for providing you with the easiest solution to produce minimal and secure images for your JVM-apps that can then be deployed unchanged on VirtualBox and AWS.

As shown in our three part article on creating a Continuous Deployment pipeline for Dropwizard-based unikernels from GitHub to AWS, we relied on Travis CI and the CloudCaptain Maven plugin to fuse and run the images.

We always strongly believed in direct build tool integration. That's also the reason why we've had Maven support from the get go. However these days a large number of you favor Gradle and its small and readable build files to build your projects. Today we are happy to announce the release of the CloudCaptain Gradle plugin. You can now use all CloudCaptain commands directly from within your Gradle build.

How does it work?

You can use the CloudCaptain Gradle plugin in a number of ways. The easiest is to simply include the CloudCaptain plugin in build.gradle and add some basic configuration to the CloudCaptain extension:

buildscript {
    repositories {
        // https is disabled until https://issues.gradle.org/browse/GRADLE-3250 is fixed
        maven { url 'https://files.cloudcaptain.sh' }
    }
    dependencies {
        classpath "com.boxfuse.client:boxfuse-gradle-plugin:1.37.0.2024"
    }
}

apply plugin: 'boxfuse'

boxfuse {
    user='your-boxfuse-client-user'
    secret='your-boxfuse-client-secret'
}

You can now execute all CloudCaptain commands as Gradle tasks directly from the command-line:

> gradle build boxfuseRun -info

As you can see there is no need to manually specify the payload as CloudCaptain will automatically default to use the project artifact. It also automatically uses the project name and version for the image.

Tip: If you do not want to store your CloudCaptain credentials directly in your build file, you can also simply store them in the BOXFUSE_USER and BOXFUSE_SECRET environment variables and they will be picked up automatically.

Build integration

Where CloudCaptain really shines, is when you integrate it directly into your build. You can then:

fuse your artifact into a new image
launch an instance of that image on VirtualBox or your CloudCaptain test environment on EC2
execute your integration tests against it
shut it down
and if successful deploy the same image unchanged to your production environment on AWS

How can you achieve all this with CloudCaptain and Gradle? All you need is this simple configuration:

buildscript {
    repositories {
        // https is disabled until https://issues.gradle.org/browse/GRADLE-3250 is fixed
        maven { url 'https://files.cloudcaptain.sh' }
    }
    dependencies {
        classpath "com.boxfuse.client:boxfuse-gradle-plugin:1.37.0.2024"
    }
}

apply plugin: 'boxfuse'
apply plugin: 'java'

//Omitted dependencies, repositories, ...

//Run deploy & run integration tests in test environment
boxfuse {
    env='test'
}
task boxfuseRunTest( type: com.boxfuse.client.gradle.task.RunTask )
task boxfuseKillTest( type: com.boxfuse.client.gradle.task.KillTask )
//Pass the url of the newly launched instance to the tests
task testConfig (dependsOn: boxfuseRunTest) << {
    project.tasks.test.systemProperty 'instanceUrl', boxfuse.instancesResult[0].url
}
test.dependsOn(testConfig)
//Always kill the test instance, even if the tests fail
test.finalizedBy(boxfuseKillTest)

//Deploy with zero-downtime on prod environment
extensions.create('boxfuseProd', com.boxfuse.client.gradle.CloudCaptainExtension)
boxfuseProd {
    env='prod'
}
//Ensure we deploy the same image to prod as the one we just tested
task boxfuseProdConfig (dependsOn: boxfuseKillTest) << {
    project.tasks.boxfuseRunProd.extension = boxfuseProd
    boxfuseProd.image=boxfuse.image
}
task boxfuseRunProd( type: com.boxfuse.client.gradle.task.RunTask, dependsOn: boxfuseProdConfig )
check.dependsOn(boxfuseRunProd)

And now all you need to do is issue

> gradle build -info

And there you have it! CloudCaptain will fuse an image, run and kill instances in multiple environments and more, all while integrated directly in your Gradle build.

As you can see, you can pass boxfuse.instancesResult[0].url as a system property to your tests, so they know where the instance under test is running. CloudCaptain dynamically sets this and others properties after it launches an instance to make it easy for you to interact with your new instance.

Tip: We extended our Dropwizard sample project with a build.gradle file so you can see it all in action and easily play with it for yourself.

Summary

In this post we looked at the CloudCaptain Gradle plugin and how it makes it a breeze to fuse images and run instances directly from your Gradle build.

You can find more information in the docs for the CloudCaptain Gradle plugin.

Have fun! And if you haven't already, sign up for your CloudCaptain account today. All you need is a GitHub user and you'll be up and running in no time. The CloudCaptain free plan aligns perfectly with the AWS free tier, so you can deploy your application to the cloud completely free.

Logging for Cloud Native Apps

2015-05-18T00:00:00+00:00

When it comes to building and running applications that are native to the cloud a number of old habits need to die and make way for modern alternatives. One of these is traditional file-based logging.

The days of using tail -f to watch logs scroll by in 5 different terminals are over.

This becomes even clearer as you move away from traditional operating systems and classic provisioning towards modern secure minimal images like those generated by CloudCaptain. These machines are tailor-made for running your application and don't contain any legacy cruft. In fact, they don't even let you log in as there is no ssh on board. All that is left, is the most efficient structure to run your application on VirtualBox and in the cloud.

So how do traditional instances compare to modern cloud native instances?

Traditional Instances	Cloud Native Instances
Long lived	Disposable
Reconfigured on change	Regenerated on change
Persistent storage	Ephemeral storage
Multiple GBs	Few MBs
Ability to log in via SSH	No SSH

Quite a different world indeed.

As you can see traditional file-based log rotation is not only outdated and risky (what if the instance's disk dies?), but also simply doesn't make sense anymore without the ability to log in.

However logging as such is still a very important tool. It is invaluable for system behavior and error root cause analysis. So how can it be done in this new cloud native world?

Logging needs

Here at CloudCaptain we view logs as great forensic tool in case of trouble. We do not see logging as a general purpose data sink. If you need structured access to large amounts of data in your logs, you should probably consider other data stores instead.

So what are the needs a logging solution must fulfill?

It must be durable for a certain duration (log value decreases exponentially with time)
It must be searchable (to quickly find related needless in the haystack)
It must be centralized (all instances of an application send their logs to the same place)
It must be integrated with the application (no mandatory clunky detours via disk or syslog)
It must be fast with little overhead (no substantial performance degradation)

Additionally we also want to be able to capture early system logs in case our application fails to start or can't connect to the centralized logging facility.

We therefore want to distinguish between instance boot logs (low volume, instance-specific, for diagnosing startup issues) and application logs (high volume, application-level, for diagnosing application behavior issues).

Instance Boot logs

The instance boot logs collect information from the earliest stages of instance startup (kernel boot) all the way to the point where the application is fully up and running. They are great for identifying environment-specific network and application configuration issues.

Luckily all virtualization and cloud platforms make this information available and if you are using CloudCaptain you have a unified and convenient way to access it. All you need to do is

> boxfuse logs instance-id

and you immediately see the boot log for that instance including DHCP lease acquiry and application startup details.

Application logs

On the application side of things there are number of excellent logging solutions available that fulfill all criteria described above. For JVM-based applications Loggly, Logentries and Papertrail make it a breeze as they all support direct Logback and Log4J integration.

Messages are sent directly by a log appender from memory via HTTPS to their backend connectors where they are ingested, aggregated, indexed and made available via nice web-based dashboards.

Let's look at one example in detail.

Integrating Logback with Loggly

Let's look at how simple it is to integrate two popular solutions: Logback (successor to Log4J) and Loggly (we could have just as easily picked Logentries or Papertrail).

Start by creating a free Loggly account and add a Java Logback source:

Now all you need to do is add the Loggly appender dependency to your build (it is available on Maven Central):

<dependency>
    <groupId>org.logback-extensions</groupId>
    <artifactId>logback-ext-loggly</artifactId>
    <version>0.1.2</version>
</dependency>

and configure the appender in your logback.xml (don't forget to make it async to avoid a performance hit):

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="loggly" class="ch.qos.logback.ext.loggly.LogglyAppender">
        <endpointUrl>https://logs-01.loggly.com/inputs/your-loggly-input-id</endpointUrl>
        <pattern>%d{ISO8601} %p %t %c{0}.%M - %m%n</pattern>
    </appender>
    <appender name="loggly-async" class="ch.qos.logback.classic.AsyncAppender">
        <appender-ref ref="loggly" />
    </appender>
    <root level="info">
        <appender-ref ref="loggly-async" />
    </root>
</configuration>

That's it! Almost immediately you will see the logs appearing in the Loggly UI:

You can now start search them, generate graphs and alerts, create custom dashboards and more!

Summary

Logging for cloud native applications is a breeze. Simply combine CloudCaptain for instance boot logs with modern hosted solutions like Loggly, Logentries or Papertrail and their Logback or Log4J integrations for your application logs and you will have a comprehensive, robust and highly available solutions up and running in no time.

Tired of snowflake servers? Create your CloudCaptain account now, start generating minimal and secure images in seconds and run them identically on VirtualBox and EC2.

Deploy a Dropwizard Unikernel to AWS - Part 3: Continuous Deployment with GitHub and Travis CI

2015-05-08T00:00:00+00:00

In part 1 of this series we looked at how to fuse a Dropwizard application into a secure unikernel image. We then tested it locally on VirtualBox and finally we deployed it unchanged to AWS.

In part 2 we automated the workflow using Maven. With every automated build, we fused an image, ran integration tests against it on VirtualBox and deployed it unchanged to EC2.

Today in the final part of this series, we take this to its natural conclusion and create a full Continuous Deployment pipeline for our unikernel. Every commit flows smoothly and automatically all the way from GitHub to AWS, using Travis CI, Maven and CloudCaptain.

Prerequisites

This post assumes you have everything set up as described in part 1 and part 2.

Step 1: Setting up GitHub and Travis CI

Here at CloudCaptain we are big fans of both GitHub and Travis CI. Together they form a powerful combo of source control and continuous integration. We especially like the fact that Travis CI integrates really well with GitHub and lets you check the whole CI build job definition straight into source control. And best of all, they are both completely free for public repositories, so that is what we'll use.

So without further ado, let's get started!

Now head over to Travis CI and sign in with your GitHub credentials. It's easy and there is no extra registration required.

Then go to your profile, sync your repositories and activate Travis CI for dwunikernel:

Finally, go to your Travis CI repository settings and define two secure environment variables for your CloudCaptain credentials called BOXFUSE_USER and BOXFUSE_SECRET. Their values are the same as the ones you enterred in your pom.xml in part 2.

The big advantage is that they'll be securely stored at Travis CI and will not appear in neither your public GitHub repository nor your Travis CI build logs, yet the CloudCaptain Maven plugin will be able to use them.

Great! That's all we need to do on this end. Do not check in anything yet. We first need to make some changes to our app.

Step 2: Tuning the build definition

Before we can check in, some important changes must be made. This is both due to the nature of public repositories and the build environment for Travis CI.

The first and most important thing you must do is delete your CloudCaptain credentials from the pom.xml. This is essential, because checking them in basically means giving the entire world access to your CloudCaptain account!

The next thing we need to take into account is that Travis CI does not support VirtualBox. This is important for running our integration test. We will therefore adjust the CloudCaptain Maven plugin configuration to use a test environment on AWS instead.

This is what the end result in the pom.xml looks like:

<plugin>
    <groupId>com.boxfuse.client</groupId>
    <artifactId>boxfuse-maven-plugin</artifactId>
    <version>1.37.0.2024</version>
    <!-- No more CloudCaptain credentials in the POM. -->
    <!-- Instead the BOXFUSE_USER and BOXFUSE_SECRET secure environment variables will be used. -->
    <executions>
        <execution>
            <id>fuse-image</id>
            <goals>
                <goal>fuse</goal>
            </goals>
            <phase>package</phase>
        </execution>
        <execution>
            <id>run-on-test</id>
            <goals>
                <goal>run</goal>
            </goals>
            <phase>pre-integration-test</phase>
            <configuration>
                <env>test</env>
            </configuration>
        </execution>
        <execution>
            <id>kill-on-test</id>
            <goals>
                <goal>kill</goal>
            </goals>
            <phase>post-integration-test</phase>
            <configuration>
                <env>test</env>
            </configuration>
        </execution>
        <execution>
            <id>run-on-prod</id>
            <goals>
                <goal>run</goal>
            </goals>
            <phase>deploy</phase>
            <configuration>
                <env>prod</env>
            </configuration>
        </execution>
    </executions>
</plugin>

We also want to give each unikernel we build a unique version. For this Travis CI makes a number of interesting environment variables available to every build. In this case we will use the TRAVIS_BUILD_NUMBER environment variable so the version of each unikernel image can be nicely traced back to the build that created it. On your project, you will probably want to use the full Maven Releases on Steroids process to also tag the source control repository.

To make this possible we will give our project a generic version (0-SNAPSHOT) and replace it dynamically at the start of every build with the Travis CI build number using the set goal of the Maven Versions plugin.

So let's first put a generic version in our pom.xml:

<project ...

    <groupId>dwunikernel</groupId>
    <artifactId>dwunikernel</artifactId>
    <version>0-SNAPSHOT</version>
    ...
</project>

Then add the Versions plugin with the new version set to TRAVIS_BUILD_NUMBER environment variable:

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>versions-maven-plugin</artifactId>
    <version>2.2</version>
    <configuration>
        <newVersion>${env.TRAVIS_BUILD_NUMBER}</newVersion>
    </configuration>
</plugin>

Finally, we must add a small YAML descriptor called .travis.yml to the root of our project to tell Travis CI how to build it:

language: java
jdk: oraclejdk8
install: mvn versions:set
script: mvn deploy

This tells Travis CI to use Java 8 and first set the version, then build, test and deploy the project.

And that's all! We are all set to fire this thing up!

Step 3: Commit, push, sit back and enjoy the show!

We are now ready. Clone your new dwunikernel GitHub repository, add all the code we created, commit and push.

All the gears will now kick into motion:

GitHub will notify Travis CI of the push
Travis CI will create a fresh build worker, check out the sources and start Maven
Maven will run once to set the version to match the build number
Maven will then run again to compile and package the code as an executable Jar
CloudCaptain will fuse that Jar into a unikernel image
CloudCaptain will then launch an instance of that image in a test environment on AWS
Maven will execute integration tests against that instance
CloudCaptain will terminate the instance
Maven will verify the test results
And finally if everything went well, CloudCaptain will update our production environment on AWS with zero downtime

And you now get all of this fully automatically after every commit!

That includes spinning up a new Travis CI worker, Maven downloading the internet, CloudCaptain fusing the unikernel image, creating an AMI, launching 2 new instance on AWS, successfully running integration tests against one of them AND performing a clean blue/green zero downtime update with the other.

As you can see the entire process takes just 5 minutes from GitHub commit to being live on AWS.

Summary

Congratulations! You now have a fully automated Continuous Deployment pipeline for Dropwizard unikernels.

It races from GitHub to AWS in 5 minutes, including the automated image build, deployment and the integration tests against the exact same image you run in production.

It is fully automated. It is reliable. It is secure. And it's fast.

And best of all, it's completely free!

So say goodbye to snowflake servers and unreliable infrastructure.
Create your CloudCaptain account now to get started with fast and reliable dropwizard unikernels on AWS.

You can find the repository we used on GitHub and the build jobs for this article on Travis CI.

Learn more:

Stay tuned for much more exciting news, subscribe to this blog and follow CloudCaptain on Twitter

Deploy a Dropwizard Unikernel to AWS - Part 2: Automated Maven build

2015-04-30T00:00:00+00:00

Today we are going to continue where we left off and automate the steps from part 1 directly from our Maven build. In addition, we are going to automatically execute an integration test against a test instance deployed on VirtualBox before pusing the image out to AWS.

Prerequisites

This post assumes you have everything set up as described in part 1.

Step 1: Automatically fusing the unikernel image

Instead of manually having to type some commands, we now automatically fuse our Dropwizard app into a unikernel image using the CloudCaptain Maven plugin as part every build. The image effectively supplants the jar file as the final product of our build.

It however shares all the essential qualities of the jar file:

one immutable artifact
regenerated after every changed
promoted unchanged from environment to environment

And it avoids the classic mistake of building a separate artifact per environment, where the artifact you run in production is different from the one you tested in test. In other words it avoids the very mistake that happens with classic provisioning of systems.

For this we first need to add the CloudCaptain Maven repository to our pom.xml:

<pluginRepositories>
    <pluginRepository>
        <id>central</id>
        <url>https://repo1.maven.org/maven2</url>
    </pluginRepository>
    <pluginRepository>
        <id>boxfuse-repo</id>
        <url>https://files.cloudcaptain.sh</url>
    </pluginRepository>
</pluginRepositories>

We now add the CloudCaptain Maven plugin after the shade plugin and define one execution in the package phase to fuse the image:

<!-- must be placed after the shade plugin to guarantee the correct execution order -->
<plugin>
    <groupId>com.boxfuse.client</groupId>
    <artifactId>boxfuse-maven-plugin</artifactId>
    <version>1.37.0.2024</version>
    <configuration>
        <user>your-boxfuse-client-user</user>
        <secret>your-boxfuse-client-secret</secret>
    </configuration>
    <executions>
        <execution>
            <id>fuse-image</id>
            <goals>
                <goal>fuse</goal>
            </goals>
            <phase>package</phase>
        </execution>
    </executions>
</plugin>

Don't forget to replace the CloudCaptain user and secret with the ones specified on your download page of the CloudCaptain Console.

Executing the build now gives us this:

> mvn clean package
...
[INFO] --- boxfuse-maven-plugin:1.37.0.2024:fuse (fuse-image) @ dwunikernel ---
[INFO] Fusing Image for dwunikernel-1.0.jar ...
[INFO] Image fused in 00:07.273s (54017 K) -> axelfontaine/dwunikernel:1.0

And there we have it! A new image generated as part of every build.

Step 2: Adding an integration test against VirtualBox

Now let's automatically

run an instance of our new image on VirtualBox in the pre-integration-test phase
execute an integration test against it in the integration-test phase
kill the instance in the post-integration-test phase
verify the results in the verify phase

Let's start by creating our integration test. For that we'll need to add junit and fluent-hc (the fluent interface for the Apache HTTP client) to the test scope of the pom.xml:

<dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <version>4.12</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>fluent-hc</artifactId>
    <version>4.3.6</version>
    <scope>test</scope>
</dependency>

Now let's add an integration test called dwunikernel.resources.HelloWorldResourceTest to src/test/java:

package dwunikernel.resources;

import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.http.client.fluent.Request;
import org.junit.Test;
import java.util.Map;

import static org.junit.Assert.assertEquals;

public class HelloWorldResourceTest {
    @Test
    public void hello() throws Exception {
        String url = System.getProperty("instanceUrl") + "?name=IntegrationTest";
        String json = Request.Get(url).execute().returnContent().asString();
        Map<String, Object> result = new ObjectMapper().readValue(json, new TypeReference<Map<String, Object>>(){});
        assertEquals("Hello, IntegrationTest!", result.get("hello"));
    }
}

As the url of the instance may change based on port availability, you can see it is being passed in dynamically using the system property instanceUrl. It is then being used to connect to the running instance, where the response of the request is then converted from a json string into a map object and compared against our expectations.

Next add additional executions of the CloudCaptain Maven plugin to the pom.xml to automatically run and kill the instance:

<execution>
    <id>run-on-virtualbox</id>
    <goals>
        <goal>run</goal>
    </goals>
    <phase>pre-integration-test</phase>
</execution>
<execution>
    <id>kill-on-virtualbox</id>
    <goals>
        <goal>kill</goal>
    </goals>
    <phase>post-integration-test</phase>
</execution>

And finally configure the Surefire and Failsafe plugins to run our test and verify the results:

<plugin>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.18.1</version>
    <configuration>
        <excludes>
            <exclude>**/*ResourceTest.java</exclude>
        </excludes>
    </configuration>
</plugin>
<plugin>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>2.18.1</version>
    <configuration>
        <includes>
            <include>**/*ResourceTest.java</include>
        </includes>
        <systemPropertyVariables>
            <instanceUrl>${boxfuse.instances.0.url}</instanceUrl>
        </systemPropertyVariables>
    </configuration>
    <executions>
        <execution>
            <id>integration-test</id>
            <goals>
                <goal>integration-test</goal>
            </goals>
        </execution>
        <execution>
            <id>verify</id>
            <goals>
                <goal>verify</goal>
            </goals>
        </execution>
    </executions>
</plugin>

As you can see we pass in the variable ${boxfuse.instances.0.url} as the value of the instanceUrl System property we referred to in out test. After executing run, CloudCaptain will automatically set the ${boxfuse.instances.0.url} to the correct url of the instance that was just started. We then use that url in our test to connect to the Dropwizard application.

Time to fire up Maven and see the results:

> mvn clean verify
...
[INFO] --- boxfuse-maven-plugin:1.3.2.576:run (run-on-virtualbox) @ dwunikernel ---
[INFO] Launching Instance of axelfontaine/dwunikernel:1.0 on VirtualBox ...
[INFO] Forwarding admin-http port localhost:50001 -> vb-fc55a898:8081
[INFO] Forwarding http port localhost:8888 -> vb-fc55a898:80
[INFO] Instance launched in 00:03.851s -> vb-fc55a898
[INFO] Waiting for Payload to start on Instance vb-fc55a898 ...
[INFO] Payload started in 00:07.025s -> https://127.0.0.1:8888
[INFO]
[INFO] --- maven-failsafe-plugin:2.18.1:integration-test (integration-test) @ dwunikernel ---
[INFO] Failsafe report directory: C:\Workspaces\dwunikernel\target\failsafe-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running dwunikernel.resources.HelloWorldResourceTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.557 sec - in dwunikernel.resources.HelloWorldResourceTest

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

[INFO]
[INFO] --- boxfuse-maven-plugin:1.3.2.576:kill (kill-on-virtualbox) @ dwunikernel ---
[INFO] Killing Instance vb-fc55a898 on VirtualBox ...
[INFO] Instance killed in 00:02.349s
[INFO]
[INFO] --- maven-failsafe-plugin:2.18.1:verify (verify) @ dwunikernel ---
[INFO] Failsafe report directory: C:\Workspaces\dwunikernel\target\failsafe-reports

Success! Our test has verified our HelloWorldResource is behaving as we expect and we have now automatically tested the exact same image we'll later run on AWS. This provides us with very strong guarantees that it will work there too.

Step 3: Deploying the tested image to AWS

After we have successfully tested our image, we are now sufficiently confident to automatically push it out to our production environment on AWS. For that we need one final addition to the list of executions of the CloudCaptain Maven plugin in the pom.xml:

<execution>
    <id>run-on-aws</id>
    <goals>
        <goal>run</goal>
    </goals>
    <phase>deploy</phase>
    <configuration>
        <env>prod</env>
    </configuration>
</execution>

And also disable the deploy plugin as we have not configured a remote repository (and our images land in the CloudCaptain Vault):

<plugin>
    <artifactId>maven-deploy-plugin</artifactId>
    <version>2.8.2</version>
    <configuration>
        <skip>true</skip>
    </configuration>
</plugin>

Finally put it all together:

> mvn clean deploy
...
[INFO] Pushing axelfontaine/dwunikernel:1.0 ...
[INFO] Verifying axelfontaine/dwunikernel:1.0 ...
[INFO] Waiting for AWS to create an AMI for axelfontaine/dwunikernel:1.0 in eu-central-1 (this may take up to 50 seconds) ...
[INFO] AMI created in 00:31.703s -> ami-842d1299
[INFO] Creating security group boxfuse-sg_axelfontaine/dwunikernel:1.0 ...
[INFO] Launching t2.micro instance of axelfontaine/dwunikernel:1.0 (ami-842d1299) in eu-central-1 ...
[INFO] Instance launched in 00:22.192s -> i-0cd66acd
[INFO] Waiting for Payload to start on Instance i-0cd66acd at https://52.28.79.74:8081/healthcheck ...
[INFO] Remapping Elastic IP 52.28.20.51 to i-0cd66acd ...
[INFO] Waiting 15s for AWS to complete Elastic IP Zero Downtime transition ...
[INFO] Terminating instance i-2ed66aef ...
[INFO] Destroying Security Group sg-fc548d95 ...
[INFO] Deployment completed successfully. axelfontaine/dwunikernel:1.0 is up and running at https://dwunikernel-axelfontaine.boxfuse.io:8081

And there we have it. We have successfully pushed the image to the CloudCaptain Vault, started a new instance on AWS and remapped the Elastic IP for a smooth Zero Downtime transition. No additional work is required. We now have this as part of every single build.

Done

Congratulations! We have taken our unikernel from part 1 and with just a few additions to the pom.xml you now have a fully automated deployment process. That includes both the dev environment, including automated testing against VirtualBox, and the production environment with Blue/Green deployments with zero downtime on AWS.

Now go ahead, dive deeper and read the Maven plugin documentation.

Continue to part 3, where we take this to 11 by creating a fully automated Continuous Deployment pipeline using GitHub and TravisCI.

Deploy a Dropwizard Unikernel to AWS - Part 1: Up and running

2015-04-25T00:00:00+00:00

Here at CloudCaptain we strongly believe in keeping things simple. We love lightweight solutions that do away with cargo cult and just work.

It turns out that when creating REST microservices Dropwizard is a perfect match for this.

In this post we are going to look at how to fuse a Dropwizard application into a secure unikernel image. We will then test it locally on VirtualBox and finally we'll deploy it unchanged to AWS. All this in 3 easy steps.

Prerequisites

Before we get started, you will need a couple of things.

First of all you will need an AWS Account. Go ahead and register for a free AWS account now if you haven't got one already. Their free tier is great and will give you a free t2.micro (1 vCPU, 1 GB RAM) instance, a few GBs of S3 storage and much more for a whole year.

You will also need a CloudCaptain Account. If you haven't got one already, create a free CloudCaptain account now by logging in the CloudCaptain Console using your GitHub account. Once logged in, connect your AWS account with your CloudCaptain account.

Finally, ensure you have the CloudCaptain Client, Java, Maven and VirtualBox installed on your machine.

This tutorial works with Windows, Mac OSX and Linux and should take about 10 minutes to complete.

Step 0: Create a Dropwizard app

Let's begin by creating a small Dropwizard application based on the latest Dropwizard Maven archetype:

> mvn archetype:generate -DgroupId=dwunikernel -DartifactId=dwunikernel -Dversion=1.0 -Dname=DwUnikernel -Dpackage=dwunikernel -DarchetypeGroupId=io.dropwizard.archetypes -DarchetypeArtifactId=java-simple -DinteractiveMode=false

This generated the basic skeleton for Dropwizard:

Now let's add a resource and a healthcheck.

For the resource create a class named dwunikernel.resources.HelloWorldResource with the following code:

package dwunikernel.resources;

import com.google.common.base.Optional;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.MediaType;
import java.util.HashMap;
import java.util.Map;

@Path("/")
@Produces(MediaType.APPLICATION_JSON)
public class HelloWorldResource {
    private final String template;

    public HelloWorldResource(String template) {
        this.template = template;
    }

    @GET
    public Map<String, Object> sayHello(@QueryParam("name") Optional<String> name) {
        Map<String, Object> result = new HashMap<>();
        result.put("hello", String.format(template, name.or("stranger")));
        return result;
    }
}

For the healthcheck, create a class named dwunikernel.health.TemplateHealthCheck:

package dwunikernel.health;

import com.codahale.metrics.health.HealthCheck;

public class TemplateHealthCheck extends HealthCheck {
    private final String template;

    public TemplateHealthCheck(String template) {
        this.template = template;
    }

    @Override
    protected Result check() throws Exception {
        final String greeting = String.format(template, "TEST");
        if (!greeting.contains("TEST")) {
            return Result.unhealthy("template doesn't include a name");
        }
        return Result.healthy();
    }
}

Finally register both the resource and the healthcheck in the dwunikernel.DwUnikernelApplication.run() method:

@Override
public void run(final DwUnikernelConfiguration configuration, final Environment environment) {
    String template = "Hello, %s!";
    environment.jersey().register(new HelloWorldResource(template));
    environment.healthChecks().register("template", new TemplateHealthCheck(template));
}

And that's it. We now have a basic Dropwizard application we can use!

Step 1: Fuse the Dropwizard app into a secure unikernel image

Log in to the CloudCaptain Console and create a new application called dwunikernel. Select the defaults of Single Instance with Zero Downtime updates (elastic IP) and the AWS eu-central-1 region:

Now build our Dropwizard app using Maven:

> mvn package

With that done you can now turn into it a secure unikernel using the CloudCaptain Client:

> boxfuse fuse target/dwunikernel-1.0.jar

Fusing Image for dwunikernel-1.0.jar ...
Image fused in 00:07.073s (54017 K) -> axelfontaine/dwunikernel:1.0

In just over 7 seconds, CloudCaptain analysed the application, determined it was a Dropwizard application running on the JVM and automatically created the smallest possible image to run it. It also automatically set the ports and the healthcheck configuration to match that of Dropwizard.

Let's check our available images:

> boxfuse ls

Images available locally:
+------------------------------+---------------------+-------+---------+------------+--------------------------------+---------+---------------------+
| Image                        |       Payload       | Debug |  Java   | AppServer  |             Ports              |  Size   |    Generated at     |
+------------------------------+---------------------+-------+---------+------------+--------------------------------+---------+---------------------+
| axelfontaine/dwunikernel:1.0 | dwunikernel-1.0.jar | false | 8.45.14 | Dropwizard | admin-http -> 8081, http -> 80 | 54017 K | 2015-04-25 23:32:54 |
+------------------------------+---------------------+-------+---------+------------+--------------------------------+---------+---------------------+
Total: 1

Congratulations! You now a have a brand new 54 MB secure unikernel image including only our application, Java 8 and the Linux kernel!

For reference, if you would have create a minimal Ubuntu Server 15.04 VM with nothing, except SSH, openjdk-8-jre-headless and our application, this is how large the image would have been:

The CloudCaptain unikernel image is just 4% of the equivalent Ubuntu one. And that's just the size. When you factor in creation time, security attack surface and reproducibility we are in two different leagues.

Step 2: Test it locally on VirtualBox

Now let's launch it on VirtualBox:

> boxfuse run dwunikernel:1.0

Launching Instance of axelfontaine/dwunikernel:1.0 on VirtualBox ...
Forwarding admin-http port localhost:50001 -> vb-033da2a4:8081
Forwarding http port localhost:8888 -> vb-033da2a4:80
Instance launched in 00:04.176s -> vb-033da2a4
Waiting for Payload to start on Instance vb-033da2a4 ...
Payload started in 00:08.525s -> https://127.0.0.1:8888

Now let's check everything is up:

> curl https://localhost:8888?name=CloudCaptain
{"hello":"Hello, CloudCaptain!"}

> curl https://localhost:50001/healthcheck
{"deadlocks":{"healthy":true},"template":{"healthy":true}}

And indeed it shows up in our list of instances as well:

> boxfuse ps

Running Instances on VirtualBox :
+-------------+------------------------------+---------------------+-----------------------+---------------------+
|  Instance   |             Image            |        Type         |          URL          |     Launched at     |
+-------------+------------------------------+---------------------+-----------------------+---------------------+
| vb-033da2a4 | axelfontaine/dwunikernel:1.0 | 4 CPU / 1024 MB RAM | https://127.0.0.1:8888 | 2015-04-25 23:42:20 |
+-------------+------------------------------+---------------------+-----------------------+---------------------+
Total: 1

Step 3: Deploy it unchanged to AWS

Finally deploy the image unchanged to AWS:

> boxfuse run dwunikernel:1.0 -env=prod

Pushing axelfontaine/dwunikernel:1.0 ...
Verifying axelfontaine/dwunikernel:1.0 ...
Creating Elastic IP ...
Mapping dwunikernel-axelfontaine.boxfuse.io to 52.28.23.63 ...
Waiting for AWS to create an AMI for axelfontaine/dwunikernel:1.0 in eu-central-1 (this may take up to 50 seconds) ...
AMI created in 00:31.067s -> ami-da744bc7
Creating security group boxfuse-sg_axelfontaine/dwunikernel:1.0 ...
Launching t2.micro instance of axelfontaine/dwunikernel:1.0 (ami-da744bc7) in eu-central-1 ...
Instance launched in 00:28.320s -> i-53be0c92
Waiting for Payload to start on Instance i-53be0c92 at https://52.28.14.76:8081/healthcheck ...
Remapping Elastic IP 52.28.23.63 to i-53be0c92 ...
Waiting 15s for AWS to complete Elastic IP Zero Downtime transition ...
Deployment completed successfully. axelfontaine/dwunikernel:1.0 is up and running at https://dwunikernel-axelfontaine.boxfuse.io:8081

CloudCaptain automatically created a new Elastic IP address and a new domain. It pushed the image to the secure CloudCaptain vault, registered an AMI, created a security group and launched an instance. Then finally it mapped the Elastic IP onto the instance.

Let's check if the instance came up:

> curl https://dwunikernel-axelfontaine.boxfuse.io?name=CloudCaptainAndAWS
{"hello":"Hello, CloudCaptainAndAWS!"}

> curl https://dwunikernel-axelfontaine.boxfuse.io:8081/healthcheck
{"deadlocks":{"healthy":true},"template":{"healthy":true}}

And it did. Our Dropwizard unikernel is successfully up and running on AWS.

Done.

Congratulations! In just a few minutes and 3 easy steps you created a brand new unikernel image of the example Dropwizard application. This image is minimal, secure and portable. You ran it unchanged on both VirtualBox and AWS.

And best of all, all of this is completely free by combining the AWS and the CloudCaptain free tiers.

Now go ahead, dive deeper and read the documentation.

Continue to part 2, where we will integrate CloudCaptain directly into the Maven build.

Welcome to CloudCaptain

2015-04-15T00:00:00+00:00

After a lot of hard work, we are excited to announce CloudCaptain!

CloudCaptain redefines the Operating System, Provisioning and Application Deployment by adopting a fresh approach designed from the ground up for a modern cloud-based world without lock-in. Images are measured in MBs and generated in seconds. Deployment are blue/green and transactional with zero downtime. Welcome to the future.

The End of the general-purpose Operating System

CloudCaptain eliminates the general-purpose operating system and replaces it with a Linux-kernel based system that is just a few MBs in size. No more bloat and unnecessary baggage. No more cargo cult. These images are as lean as they get.

A CloudCaptain image is just 1% of the size of a legacy general-purpose operating system. It is faster to transfer, cheaper to archive and, by drastically reducing the attach surface, much more secure.

The End of Traditional Provisioning

CloudCaptain eliminates traditional provisioning by generating bootable images for your applications in seconds. No more Chef and Puppet. Instead CloudCaptain analyses your app and generates the perfect immutable image for it, in seconds, every single time.

CloudCaptain image generation is on average 100x faster than traditional provisioning. Instead of taking hours or minutes, generating an image takes just 5 seconds. You don't need sysadmin knowledge and you don't have to write complex recipes. All it takes is one command. And yes, it runs on Windows, Mac & Linux.

Application Deployment Reinvented

CloudCaptain redefines application deployment and replaces it by a clean bootable Image-based model, where every Image is rolled out using a highly reliable transactional blue/green deployment with a Zero Downtime transition.

CloudCaptain deploys Images to ensure all systems are identical in all environments, down to the last bit. This is eliminates a common source of errors. Once generated you can now run the exact same Image locally in development on VirtualBox as you you will run in production on AWS.

Deployment on VirtualBox takes just a few seconds, and a Zero-Downtime update on AWS usually completes in under 90 seconds.

No more Lock-in

CloudCaptain eliminates lock in by working with your applications in their native formats and your AWS account. Say goodbye to walled gardens and proprietary formats.

At CloudCaptain we strongly believe you should be able to pick the best tool for the job without needing to rewrite your application or having to constrain yourself to a walled garden. We built CloudCaptain to help you solve the heavy lifting of deployment using the open technologies and platforms you love.

There is no lock-in and there are no long term commitments. You can leave anytime you like, but you won't want to.

Get Started

CloudCaptain is available now.

We are launching with support for JVM apps packaged as either war files on Tomcat or executable jar files. You can generate CloudCaptain micro Images on Windows, Mac or Linux. And you can run them unchanged on VirtualBox and AWS.

We created a free tier that maps perfectly onto AWS' free tier, so you can get started at no cost.

Stay tuned, subscribe to this blog by email or RSS, and follow @boxfuse. We have many exciting announcements to come. This is just the beginning!

CloudCaptain

Boxfuse is now CloudCaptain!

Transition and Improvements

Introducing CloudCaptain

Spring Boot 2 and t3 and r5 instances support

Spring Boot 2 support

T3 and R5 instance support

Available today

Introducing FastAMI

Introducing CloudCaptain FastAMI

Available today

Introducing Private Vault

Introducing Private Vault

Migrating from the CloudCaptain Vault to your Private Vault

Available today

Launch M5 and C5 instances on the Nitro System

Full CloudCaptain support

Available today

Introducing load-balanced-https apps

How does it work?

Why Application Load Balancers?

Seeing it in action

BYOTG (Bring Your Own Target Group)

Available today

Customer Interview: KwiqJobs

Hi Jannes, can you tell us about yourself and your company?

That sounds great! What does your infrastructure setup look like?

Why did you choose CloudCaptain?

Can you tell us a bit more on how you are using CloudCaptain and what your favorite feature is?

Awesome! Can you share something about your plans for the year ahead?

Thanks!

One-off apps

Introducing one-off apps

How does it work?

Available today

Hello Hyper-V!

How it works

Available today

Add Custom Tags to your AWS Resources

Introducing Dead-Easy Custom Tags

Typical Uses

How it works

Available today

Automatic TLS (SSL) Certificates

How does it work?

Seeing it in action

Obtaining our certificate

Creating the app

Running our application

Available today

Custom Elastic IPs

How does it work?

Using the CloudCaptain Client

Available today

Custom Domains

How does it work?

Using the CloudCaptain Client

Available today

Shipping native Linux x64 binaries and libs with JVM apps

How does it work?

Seeing in action

Available today

Logback and Log4J2 appender for AWS CloudWatch Logs

Installation

Maven

Gradle

Usage

Logback

Log4J2

Code

Displaying the Logs

Live tailing

Log filtering

Available today

Deploy any Linux x64 app on AWS with one command

Seeing it in action

Summary

Monitoring with New Relic

Server Monitoring

Installation