Cloud Foundry Buildpacks

A buildpack provides framework and run time support for applications. Buildpacks examine user-provided artifacts to determine what dependencies to download and how to configure applications to communicate with different services.

When we push an application, Cloud Foundry automatically detects which buildpack is required and installs it on the Droplet Execution Agent where the application needs to run.

There are three elements of a buildpack. They are detect, compile and release. 

Detect

A script to detect condition to run buildpack. Runs this detect script against deployed application. OK if exit 0. To get an idea, let's look into the detect script of ruby buildpack.

#!/bin/bash


if [ -f "$1/Gemfile" ]; then

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
echo "ruby $(cat $SCRIPT_DIR/../VERSION)"
exit 0
else
echo "no"
exit 1
fi

If we analyse the script, what it does simply is doing a conditional check to see if there is a Gemfile. If so, ruby buildpack is used since the application is confirmed by the 'detect' script.

Detect can be written in ruby or bash.

There are some examples that detect itself is written in bash and then calls python script. cf-php buildpack is similar to that.

#!/bin/bash


BP=$(dirname $(dirname $0))
export PYTHONPATH=$BP/lib
VERSION=`cat $BP/VERSION`
python $BP/scripts/detect.py $1 $VERSION

Compile

This is the key script to setup language or framework runtime setup. 

What this does is, it downloads execution environment sources and then compiles. 

Because it takes time to do compile, many buildpacks download pre-compiled binaries.  

Release

A script to return metadata required for execution in yaml format.

#!/usr/bin/env 
ruby require 'pathname' 
release = Pathname.new(ARGV.first).join("tmp/heroku-buildpack-release-step.yml")
puts release.read if release.exist?

Staging

Staging refers to building the droplet, which is the executable bundle of application. At this time, user deploys the application, process the application by buildpack description and then create the droplet. Let me describe this graphically. 

1. When user gives the command 'cf push', user uploads the application to Cloud Controller. 

2. Cloud controller sends the 'staging.start' message to DEA to request staging activity. 

3. DEA does staging and build the droplet.

4. Once droplet is built, it returns back to cloud controller. 

The above diagram explains the complete scenario.

Admin buildpack

This is a buildpack which is directly installed in Cloud Foundry. Suppose that user did not specify a buildpack. Then Cloud Foundry selects the appropriate buildpack from the admin buildpack. Administrator can add using create-buildpack. Cloud Foundry service provider can add supported languages to the platform.

Staging.start is received by DEA.

Download application from Cloud Controller.

Execute admin buildpack detect based on priorities.

Run buildpack compile which matched.

Download Ruby binaries, setup binaries, execute 'bundle install', replace database.yml(in case of rails) etc.

Execute release and produce necessary information.

Droplet contents.

In case of a Ruby app, the droplet produced consists ruby executable created by the buildpack, uploaded application, files installed by bundle install and start command description. 

System Buildpacks

This table describes Cloud Foundry system buildpacks. Each buildpack row lists supported languages and frameworks and includes a GitHub repo link. Specific buildpack names also link to additional documentation.

Name	Other Supported Languages and Frameworks	GitHub Repo
Java	Grails, Play, Spring, or any other JVM-based language or framework	Java source
Ruby	Ruby, Rack, Rails, or Sinatra	Ruby source
Node.js	Node or JavaScript	Node.js source
Binary	NA	Binary source
Go	NA	Go source
PHP	NA	PHP source
Python	NA	Python source
Staticfile	HTML, CSS, or JavaScript	Staticfile source

Other Buildpacks

If a certain application uses a language or framework that Cloud Foundry buildpacks do not support, we can try the following ways.

1. Customize an existing buildpack, or create your own custom buildpack.

For example, by default Cloud Foundry does not support Haskell. Using a buildpack for Haskell allows us to add support for it at the deployment stage. When we push an application written using Haskell, the required buildpack is automatically installed on the Cloud Foundry Droplet Execution Agent (DEA) where the application will run.

A common development practice for custom buildpacks is to fork existing buildpacks and sync subsequent patches from upstream.

2. Use a Cloud Foundry Community Buildpack.

3. Use a Heroku Third-Party Buildpack.

Heroku maintains a collection of officially supported buildpacks but there are also third-party buildpacks that enable us to use languages and frameworks beyond those that are officially supported by Heroku.

Custom buildpack

User can specify the buildpack at the time of deployment. This is called custom buildpack. When custom buildpack is specified,  detect step is skipped, compile step is executed. 

Binary buildpack

This is a special buildpack that I came across. This is a Cloud Foundry buildpack for running arbitrary binary web servers.

Unlike most other Cloud Foundry buildpacks, the binary buildpack must be specified in order for it to be used in staging your binary file. You can specify the buildpack your app should use with the -b option for cf push as follows.

cf push my_app -b https://github.com/cloudfoundry/binary-buildpack.git

There are two ways to provide Cloud Foundry with the shell command to execute your binary.

1. Procfile

Include in your app's root directory a Procfile that specifies a web task. Eg: web: ./app

2. Command line

Alternatively, you can provide the start command when deploying your app with cf push by including a -c option:

cf push my_app -c './app' -b binary-buildpack

Container to run application

Warden

Warden is a container to isolate user application. User application runs inside Warden container. When the application stops, then the container gets destroyed. 

Currently, in addition to run the application, staging process itself and also used to run component in Bosh-lite environment. 

dea_next uses Warden Client to invoke Warden Server API. Communication between Warden is determined by Warden protocol. Warden Server is written in Ruby and and the container which Warden Server runs is written in C.

Why Warden(vs Docker)

• Docker only runs a single process, it does not conform Cloud Foundry architecture. 
• Cannot control limit after the container creation.
• Cannot control disk and bandwidth limits.

The main purpose is ultimately the same in Docker and Cloud Foundry.