Knowledgebase

There are many ways to automate the process of setting up and configuring a box. For whatever reason, if our whole system at this point comprises of just a single box, setting up a full SCM (Software Configuration Management) infrastructure is overkill. Shell scripts are one option, but we could also use a stripped-down version of SCM which is available in a few of the tools out there. Chef is one of the popular options and "chef-solo" is Chef's standalone configuration mode where we don't require an extra node to act as a "chef-server". All it needs is a URL or a path to a tarball package that contains chef cookbooks. Compared to shell scripts, this type of approach is more declarative and efficient out of the box and is also a good introduction to get started with SCMs or IaC (Infrastructure as Code) processes.

A few other benefits to using chef-solo:

• Composability: Use the community cookbooks from chef supermarket or other places.
• Free and open source; licensed under the permissive Apache 2.0 License.
• Access to the rest of Chef's ecosystem (InSpec, ChefSpec, Cookstyle, Foodcritic, chef-shell etc)
• The cookbooks and recipes can be later on adapted to a client/server mode.

And some downsides:

• Some community cookbooks on the Chef supermarket are outdated, broken and not maintained.
• chef-solo cannot resolve dependencies on its own.

The 'recipes' inside of a chef 'cookbook' have a ruby based DSL that describes 'resources' to be in a particular state on a node. Let's proceed with a walkthrough to get acquainted with a few Chef concepts that are also applicable to chef-solo. Our goal is to set up an Ubuntu node running a Python/Django web app using Gunicorn and NGINX.

Note: We do not necessarily require ChefDK to be installed on our "Chef workstation" (our machine), although with it, we can use 'chef generate' commands to start-off with a directory structure for creating cookbooks, recipes and more. In this article, we will assume ChefDK is installed on our workstation. Commands were run using the version 4.7.26-1 of ChefDK.

---

(Everything from this point onwards, unless specified otherwise, is to be run on our machine, also referred to as the 'Chef Workstation')

Creating the cookbook

Cookbooks in chef are a reusable units that contain everything needed to support a configuration scenario. Cookbooks can contain multiple 'recipes' and 'recipes' mostly consists of resource patterns. default.rb is the default recipe that will be run when the cookbook is referenced in a run-list. Different recipes allow for separation of concerns. For this tutorial, however, we'll add all resource declarations in one main recipe file, which is the default.rb.

Create a folder named "my-chef-project" and create a folder inside it called "cookbooks". From ./my-chef-project/cookbooks/, run:

$ chef generate cookbook my-cookbook

Our directory structure will now look like this:

.
└── my-chef-project
└── cookbooks
└── my-cookbook
├── CHANGELOG.md
├── LICENSE
├── Policyfile.rb
├── README.md
├── chefignore
├── kitchen.yml
├── metadata.rb
├── recipes
│ └── default.rb
├── spec
│ ├── spec_helper.rb
│ └── unit
│ └── recipes
│ └── default_spec.rb
└── test
└── integration
└── default
└── default_test.rb

Adding packages

The first step to setting up our node is to identify what packages are required by our app. Our node is selected to be Ubuntu, so we can rely on the APT package manager to gather the dependencies. Installing the packages provided by the OS distribution is then a piece of cake:

apt_update
package 'python3'
package 'python3-pip'
package 'nginx'
package 'pkg-config'
package 'libcairo2-dev'
package 'libjpeg-dev'
package 'libgif-dev'
package 'libgirepository1.0-dev'

These are pretty much self-explanatory. The first line will update the apt repository and the following lines will install those packages.

Note: The packages following 'nginx' are needed for compiling some of the python dependencies through pip. These may differ based on your python/django project dependencies specified in requirements.txt. You can use a trial and error method to determine these packages that you need to include in your cookbook. To do that, perform a manual sudo pip install -r requirements.txt (Note: This installs packages system wide!) on a freshly instantiated ubuntu machine to see if it runs successfully. If not, the stderr should give you hints on what packages are missing.

Creating linux users

Once we're done adding the required packages, we need to create a non-privileged Linux user that will own the application source code.

user 'bob' do
  uid 1212
  gid 'users'
  home '/home/bob'
  shell '/bin/bash'
  password '$1$alilbito$C83FsODuq0A1pUMeFPeR10'
end

Note that the password is a shadow hash format used in Linux. It can be derived using OpenSSL:

$ openssl passwd -1 -salt alilbitof mypassword

Including the app source

Now let's include the Django application source code to our cookbook. Place the source code inside ./my-chef-project/cookbooks/my-cookbook/files/default/myapp/ Create the ./my-chef-project/cookbooks/my-cookbook/files/default directory if it doesn't exist.

Instruction to copy these files to a remote location on our node is described using the remote_directory resource:

remote_directory '/home/bob/myapp' do
  source 'myapp' # This is the name of the folder containing our source code that we kept in ./my-cookbook/files/default/
  owner 'bob'
  group 'users'
  mode '0755'
  action :create
end

Pulling in the python dependencies

To install the python packages in requirements.txt, we can use the execute resource to run an arbitrary command. In this case, we need to execute the pip install command over it:

execute 'install python dependencies' do
  command 'pip3 install -r requirements.txt'
  cwd '/home/bob/myapp'
end

Note: Bear in mind that this is going to execute as the root user and the python libraries will be installed system-wide. If our node is designated to exclusively run this one single python app, then it isn't much of a problem. Despite that, a better option to keep things clean and sane is to find and use a community cookbook that manages python installations or 'virtualenvs'. (or at the very least, write a series of execute blocks to replicate this). Using virtualenvs in python ensures that any python based system tools or other python projects will not be affected

Setting up Gunicorn & NGINX

Now it's time to prepare the Gunicorn WSGI HTTP Server with NGINX as our reverse proxy. Nginx is also used to handle all the static assets from Django.

To strap up Gunicorn as a service on Ubuntu, Systemd can be used. The systemd_unit resource is included in Chef since version 12.11.

systemd_unit 'gunicorn.service' do
  content({
  Unit: {
    Description: 'Django on Gunicorn',
    After: 'network.target',
  },
  Service: {
    ExecStart: '/usr/local/bin/gunicorn --workers 3 --bind localhost:8080 myapp.wsgi:application',
    User: 'bob',
    Group: 'www-data',
    WorkingDirectory: '/home/bob/myapp'
    Restart: 'always',
  },
  Install: {
    WantedBy: 'multi-user.target',
  }
  })
  action [:create, :enable, :start]
end

Now we have to include a standard NGINX proxy configuration to this Gunicorn server as shown below. This snippet can go into ./my-cookbook/templates/nginx.conf.erb. Create the templates directory if it doesn't exist.

Note: Chef's templates support embedded ruby files that can contain variables, ruby expressions and statements. Although this file has the 'erb' extension, we did not use any of the ruby statements or expressions. Also, for the sake of simplicity, we only have a non HTTPS nginx config here (gentle reminder; please do not do this in production!)

server {
  listen 80;
  server_name http://example.com/;

  location = /favicon.ico { access_log off; log_not_found off; }
  location /static/ {
    root /home/bob/myapp/myapp/static;
  }

  location / {
  include proxy_params;
    proxy_pass http://localhost:8080/;
  }
}

Note: There is also an alternative and a better config, where, for instance the Gunicorn server is bound to a unix domain socket instead of a TCP loopback connection. It's worth exploring that for performance reasons.

To copy over this config to sites-enabled folder on the node, use the template resource from Chef.

template '/etc/nginx/sites-available/example.com.conf' do
  source 'nginx.conf.erb'
  owner 'root'
  group 'root'
  mode '0744'
end

Activating configs on nginx is normally done by creating a symlink pointing to the config at sites-available in the nginx's sites-enabled folder. Symlinks can be declared in chef cookbooks with the link resource as show below:

link '/etc/nginx/sites-enabled/example.com.conf' do
  to '/etc/nginx/sites-available/example.com.conf'
end

and to delete the default config symlink:

link '/etc/nginx/sites-enabled/default' do
  action :delete
end

Starting NGINX

And finally, to fire up nginx service:

service 'nginx' do
  action :enable
  action :start
end

Runlists

Run-lists in chef are an ordered list of roles or recipes in a cookbook that will be executed in sequence on the node. We have one cookbook "my-cookbook" and the "default" recipe inside it that we need to execute on the Ubuntu box, so the runlist.json in our project directory (./my-chef-project/runlist.json) should look like this:

{
  "run_list": [
    "recipe[my-cookbook::default]"
  ]
}

Final steps

Our cookbook for Chef solo is ready to be served. It's time to provision an Ubuntu 18.04 machine and install ChefDK on it:

$ ssh root@example.com 'apt-get update && yes | apt-get install curl && curl https://packages.chef.io/files/current/chefdk/4.7.45/ubuntu/18.04/chefdk_4.7.45-1_amd64.deb -o chefdk.deb && yes | dpkg -i chefdk.deb && rm chefdk.deb'

Going back to our Chef workstation, all we need to do is put the cookbooks folder inside a tarball, transfer that tarball along with the runlist.json to the remote node we provisioned above and run the chef-solo command:

(The below command is to be run inside the node or the 'chef client' and not the Chef Workstation)

$ chef-solo --recipe-url $(pwd)/chef-solo.tar.gz -j $(pwd)/runlist.json --chef-license=accept

Or here's a one-liner (To be run from ./my-chef-project/ CWD on Chef Workstation):

tar zvcf chef-solo.tar.gz ./cookbooks &&\
scp chef-solo.tar.gz runlist.json root@example.com:~/ &&\
ssh root@example.com 'chef-solo --recipe-url $(pwd)/chef-solo.tar.gz -j $(pwd)/runlist.json --chef-license=accept'

That's it! Watch the standard output fill up with Chef activity trying to converge your node to what you've specified in the cookbooks. Chef-solo will install all the gems required for all the cookbooks. If the chef-solo command is successful, we will have a working Django application running behind nginx on the Ubuntu box. Navigate to the domain/IP to test it.

Note: Remember that in django you may need to set this domain/ip in the ALLOWED_HOSTS list in settings.py.

Responding to changes

Whenever we make a change in the content of our project directory (recipes, templates or the application source code etc), simply run the above one-liner from the project directory.

Tip: If the cookbook is version controlled with git (as it should), one good recommendation is to set git hooks to run this one-liner.

Hosting the tarball (optional)

If you look closely at the last chef-solo command, notice that the --recipe-url is meant to take a URL. This means that you can have a workflow where a CI will build your chef-solo tarball, upload it someplace and configure your node to pull from it periodically.

Tip: Use curl to pull the changed tarball periodically as a cronjob. curl -z $file will honor If-Modified-Since headers and will only download the tar ball if the remote file has been changed since the timestamp on the existing local $file.