In this post, I will introduce a really fast deployment tool - Mina, and show you how to deploy a rails application which runs on unicorn with nginx, also I’ll show you how to organize your mina deployment tasks.
“Mina is a readlly fast deployer and server automation tool”, this how the team who built it describes it. The concept behind Mina is to connect to a remote server and executes a set of shell command that you define in a local deployment file(config/deploy.rb). One of its outstanding feature is it is really fast because all bash instructions will be performed in one SSH connection.
Init
To use mina automating deployment, you need to get the following things ready.
a remote sever
create a user for deployment (e.g. deployer) and add this user into sudoer list.
generate a ssh key pair, add the generated public key into GitHub.
create a deployment target fold on the remove server(e.g. ‘/var/www/example.com’)
once you got these things done, run mina init in your project directory, this will generate a deployment file - config/deploy.rb, then set the server address, deployment user, deployment target and other settings in deployment file, like the following:
set :user, 'deployer'
set :domain, ENV['on'] == 'prod' ? '<prod ip>' : '<qa ip>'
set :deploy_to, '/var/www/example.com'
set :repository, 'git@github.com:your_company/sample.git'
set :branch, 'master'
Setup
Then run mina setup, this will create deployment folders, which looks like this:
/var/www/example.com/ # The deploy_to path
|- releases/ # Holds releases, one subdir per release
| |- 1/
| |- 2/
| |- 3/
| '- ...
|- shared/ # Holds files shared between releases
| |- logs/ # Log files are usually stored here
| `- ...
'- current/ # A symlink to the current release in releases/
Provision
It is very common to setup a new server and deploy application on it, it will be good if we can automating this process, here comes the provision task:
to be able run this taks multi times, I create some helper method detecting whether an executable exists or not. e.g. ruby_exists, exits('gem') these helper method will return if the executable exits, otherwise, it will run next command to install the executable.
With this task, you can get a server ready for deployment in seveural minutes.
Deploy
Once the server is provisioned, you can deploy your application with mina deploy, here is a typical deploy task:
Unicorn and nginx
to run our application on unicorn and nginx, we need to create our own unicorn and nginx configuration file and start nginx and unicorn with our configuration. here is the task:
Conclusion
With these tasks: mina init, mina provision, mina deploy, mina helps you deploying easily with less mistake, have fun with mina!
Recently I’m working a RoR stack RESTful API project, I was involved in proposing tech stack, architecture, deployment pipeline. There was many thoughts in this progress, so I wrote down our thoughts here, in case it may help you when you met a similar question.
Tech Stack
There’re bunch of api framework out there in ruby stack, such as Grape, rails-api, Sinatra. I’ll share my understanding here:
Sinatra
Sinatra providing a lightweight, simple DSL to creating web application. we can create a web/api application with sinatra within 15 seconds! the downside is, it is not a full stack framework, it requires us to combine sinatra with other frameworks. for example, we have a backend database for storing and retrieving information, you need to interating sinatra with a orm framework(e.g. ActivateRecrod or DataMapper).if we want to render info on a web page, we need to integrate a view template engine.
Grape
Grape is a ruby framework, which is designed for creating RESTful api service. it have sevural great features for creating RESTFul api. for example, api verisoning, automatic api doc generation, etc. Similar to sinatra, it is not a full stack framework, it requires some integration work. BTW, Grape can be easily integrated into any Rack application(e.g. sinatra, rails).
Rails::API
Rails::API is a subset of a normal Rails application, created for applications that don’t require all functionality that a complete Rails application provides. It is a bit more lightweight, and consequently a bit faster than a normal Rails application.
In the end, we choose Rails::API as our tech stack for the following reason:
it is a fullstack framework, including ORM, db migration, validation, etc all in one place.
we can leveage some rails’s feature, e.g. generator, db migration.
it is a subset of rails, is designed for creating a API application.
rails’s REST conversion.
API Design
Content Type Negotiation
A most important part of designing RESTful API is content-type negotiation. the content type negotiation can be both in request/response header and url suffix:
in request header, Content-Type indicating content type of data in request body, Accept telling server what kind of content type the client expected to accept.
in response header, Content-Type indicating content type of data in the response body.
Also, request to /users/2.json expecting the server return user info in JSON format, but request to /users/2.xml expecting get a XML response.
there’re several kind of standard content type, e.g. application/json, application/xml.
People can define their own content type, e.g. application/vnd.google-apps.script+json
My feeling is, if your api is a public endpoint, you’d better define your own content type.
let’s take a example, a authentication api expecting get the following request data:
you have two content type choices: application/json and application/vnd.mycompany.credential+json, if this is a public api, I’ll chose the customized content type - application/vnd.mycompany.credential+json, or this is an internal api, I’ll chose the standard content type - application/json. I made this choice by considering the following reasons:
Pros
Cons
Customized content type
Could define a json schema, api server and client could use this json schema ensure request is processable.
adding complexity
Standard content type
Simple and straightforward
no validation to request data, any unexpected message could be send to server
Code conversion
I struggled with workflow management when I play the very first story in this project, the problem is, it is very common and business workflow have more then two exit points. e.g. a login workflow, the possiable exit points are:
username password matched, login succeed
username password mimatched, login failed
user is inactived, login failed
user is locked because of too many login failures, login failed.
…
in a rails project, it is very important to keep your controllers clean, controller’s only responsibility is passing parameters and routating, so it is better is to put these business logic into Model or Service. here comes the problem: how can we let the controller konw the extact reason of failure without put the business logic in controller? return value can not fufill this requirement, so here come out our solution: modeling these exception exit point with ruby exception, handing different exceptions in controller with different exception handler. and we found it makes the controller and model much more cleaner. let’s have a look at this example:
Before, the controller was messy:
After, controller is much cleaner
Error code
It is very common to return failure reason when API call failed. Even we can return failure reason in plian english, as an API provider, we shouldn’t assume that API client will use error message we provided, it’s better to return a structured failure reason which can be parsed by API client. let’s take a look at example from Github API:
failure reason was returned as an JSON object, resource representing what kind of resource is requested, field indicates which field fails api call, code indicating the exact faliure reason.
Another thing I want to highlight is - do not define numeric error code, it will be a nightmare for you and your client. a better solution is define meanningful error code, like missing_field, too_long, etc.
Documentation
RESTful api don’t have frontend, so it is very important to make your document friendly. Aslo, it is very common that a developer changing code but forgot to change api doc, so it would be great if we can generating api document automaticly. considering we have a well formed test suite(or spec) for the api, why cann’t we just extract information from these tests/specs and generating document automaticly. Acturally there’re some gems trying to solve this problem: apipie-rails, swagger-ui.
we’re using apipie-rails, but we’ve found some missing features in apipie-rails. e.g. it can not record extract request and response headers, while headers play a important rule in a RESTful api.
Testing
We have two kind of tests in this project: integration test and unit test.
integration tests test the api from end point, it is a end to end test. we use requesting rspec define this test.
unit tests test different layer. hints: only stub method on the boundary of each layer.
integration test
Make tests less flakey by using contract-based testing instead of hitting live endpoints
Versioning
we could specify api version in either url or http request header. In theory, verion number in url is not restful, any thoughts, please let me know
Provision:
creating new node from a customized AMI (with many required dependence installed).
Build pipeline:
the CI will build a rpm package one all test passed, then this package will be pushed to S3, after that, this package will be installed on the provisioned node.
def main(m: M): Z = {
val x: X = pre(m)
val y: Y = sub(m, x)
val z: Z = post(m, x, y)
return z
}
def sub(m: M, x: X): Y {
val y: Y = substuff(m, x)
return y
}
def main(m: M) {
val x: X = pre(m)
val z: Z = sub(m, x, {post(m, x, _)})
return z
}
def sub(m: M, x: X, subCont: Y => Z) {
val y: Y = substuff(m, x)
val z: Z = subCont(y)
return z
} 当把包含`post`代码段传递给`sub`时,Scala生成一个记录了包含`post`执行上下文(当时的m,x值)的闭包,当这个闭包将来被执行时,可以获得的当时的执行上下文。
def prog(m: M) {
val z: Z = main(m)
println(z)
System.exit(z.exitValue)
} 现在,我们可以对`prog`和`main`做同样的CPS变换:
def prog(m: M) {
main(m, {z: Z => {
println(z)
System.exit(z.exitValue)
}})
}
def main(m: M, mainCont: Z => Unit): Unit = {
val x: X = pre(m)
val z: Z = sub(m, x, post(m, x, _))
mainCont(z)
} 我们仍然在`sub`中使用了`return`,并且在`main`中使用`sub`的返回值。为了解决这个问题,我们需要把`mainCont`做为一个`continuation`传递给`sub`,修改后的`main`和`sub`如下:
def main(m: M, mainCont: Z => Unit): Unit = {
val x: X = pre(m)
sub(m, x, y: Y => {
val z: Z = post(m, x, y)
mainCont(z)
})
}
def sub(m:M,x:X, subCont: (Y) => Unit) {
val y:Y = substuff(m,x)
subCont(y)
}
shift被用于标识delimited continuation的下界,shift内部的代码不是CPS代码,(but it’s untransformed return value is CPS)但是它的未转换的返回值是CPS。当shift执行时,会被传入一个传入一个从它的调用者开始到一个闭合的reset的continuation做为参数。
当你看到一个shift块,并且它的返回值被应用在一个表达式中时,比如上面的那个“shift + 1”的例子,请记住一点,由于代码的转换,这个从shift块中“return”从来没有真正地return过。实际上,当执行到shift块时,shfit块之后的代码被转换成一个continuation并做为参数传递给这个shift块;如果shift块中的代码执行了这个continuation,这个continuation的执行结果就表现为shfit块的返回值。the value which is passed as an argument to the continuation appears as the value being returned from the shift block.因此,传递给shift块中continuation 函数的参数的类型和代码中shfit块的返回值类型是一致的。 continuation 函数的返回值类型和这个shift块外围的reset块的返回值类型是一致的Thus the type of the argument passed to the shift block’s continuation function is the same as the type of the return value of the shift in the source code, and the type of the return value of that continuation function is the same as the type of the return value of the original last value in the reset block that encloses the shift block.
这里有三种类型与shift相关连:
传递给continuation函数的参数类型,和shfit块的语义返回值类型一致。
The type of the argument to pass to the continuation, which is the same as the syntactic return type of the shift in the source code.
The type of the return from the continuation, which is the same as the return type of all of the code that follows the
shift block in the source code (i.e. the type of the last value in the block of code between the shift block and the end of the function or reset block containing the shift block). This is called the untransformed return type.
shift块中最后一条语句的值,被做为整个函数或者reset块的返回值的类型。这个被称为transformed return type
The type of the last value in the shift block, which becomes the type of the return value of the enclosing function or return block. This is called the transformed return type.
在shift的方法签名中,上面的三个类型标记为A,B,C:
def shift[A, B, C](func: ((A => B) => C)): A @scala.util.continuations.cpsParam[B, C]