SparkleFormation generator CloudFormation templates with rainbows and unicorns

If you seriously are using the AWS (Amazon Web Services), you probably know about the ability to describe the infrastructure using JSON templates. In AWS, this service is called CloudFormation. In fact, this solution allows you to describe the desired state of any resource available in the AWS (instances, layers, opsworks, ELB, security groups, etc.). A set of resources called a stack. After download the CloudFormation template, the system either will create the necessary resources in the stack if they aren't, or try to upgrade the existing to the desired state.

This works well if you have a small amount of resources, but once the infrastructure is growing, the problems:
the

the
• In JSON is not possible to use loops and similar resources we have to repeat the same parameters and in the event of a change too (not DRY)
the
• To the configuration entry for cloud-init need double-escaping
the
• In JSON and it has a bad man-sitemost

To avoid these problems, engineers of the Heavy Water written in ruby DSL and CLI for generating and working with these templates called SparkleFormation (github).

the

DRY

When I arrived at my current project we had a CloudFormation template, containing about 1500 lines of resource description and 0 comments. After using SparkleFormation pattern began to take 300 lines, many of which are comments. How are we made? Let's see how CloudFormation works, a typical description of the resource looks like this:

ELB

 "AppsElb": {
"Type": "AWS::ElasticLoadBalancing::LoadBalancer",
"Properties": {
"Scheme": "internal",
"Subnets": [
{"Ref": "Subnet1"},
{"Ref": "Subnet2"}
],
"SecurityGroups": [
{"Ref": "SG"}
],
"HealthCheck": {
"HealthyThreshold": "2",
"Interval": "5",
"Target": "TCP:80",
"Timeout": "2",
"UnhealthyThreshold": "2"
},
"Listeners": [
{
"InstancePort": "80",
"LoadBalancerPort": "80",
"Protocol": "TCP",
"InstanceProtocol": "TCP"
},
{
"InstancePort": "22",
"LoadBalancerPort": "2222",
"Protocol": "TCP",
"InstanceProtocol": "TCP"
},
{
"InstancePort": "8500",
"LoadBalancerPort": "8500",
"Protocol": "TCP",
"InstanceProtocol": "TCP"
}
]
}
}

Because SparkleFormation allows you to use normal ruby code in DSL, then rewrite it:

Create an ELB in SparkleFormation

 resources(:AppsElb) do
type 'AWS::ElasticLoadBalancing::LoadBalancer'
properties do
scheme 'internal'
subnets [PARAMS[:Subnet1], PARAMS[:Subnet2]]
security_groups [ref!(:SG)]
# port mapping of 80>80, 22 - > 2222, etc.
listeners = { :'80' => 80, :'2222' => 22, :'8500' => 8500 }.map do |k, v|
{ 'LoadBalancerPort' => k.to_s,
'InstancePort' => v,
'Protocol' => 'TCP',
'InstanceProtocol' => 'TCP' }
end
listeners listeners
health_check do
target 'TCP:80'
healthy_threshold '2'
unhealthy_threshold '2'
interval '5'
timeout '2'
end
end
end

As you can see we are no longer repeated in the description of each port and the add new will take us only one line. Moreover if we need to create a lot of almost similar resources, but different 1-2 parameters, SparkleFormation provides such an entity as dynamics, where you can describe an abstract resource, which is passed the parameters:

Example from the documentation

# dynamics/node.rb
SparkleFormation.dynamic(:node) do |_name, _config={}|
unless(_config[:ssh_key])
parameters.set!("#{_name}_ssh_key".to_sym) do
type 'String'
end
end
dynamic!(:ec2_instance, _name).properties do
key_name _config[:ssh_key] ? _config[:ssh_key] : ref!("#{_name}_ssh_key".to_sym)
end
end

And then we can call this an abstract resource in the template:
the

SparkleFormation.new(:node_stack) do
dynamic!(:node, :fubar)
dynamic!(:node, :foobar, :ssh_key => 'default')
end

Thus we can reuse the resources we need and, if necessary, changes to change all at 1 place.

the

Cloud-init

We often use the opportunity to convey to the instance at boot the cloud-init config in a yaml file and use it to install packages, configuration, CoreOS, individual services and other settings. The problem is that yaml must pass the instance in the user-data in the CloudFormation template, and it looked like this:
"Fn::Base64": { "Fn::Join": [ "", [ "#cloud-config\n", "\n", "coreos:\n", "etcd:\n", "discovery: ", {"Ref": "AppDiscoveryURL"}, "\n", "addr: $private_ipv4:4001\n", "peer-addr: $private_ipv4:7001\n", "etcd2:\n", ...