Friday, November 23, 2012

Python, Flask, MongoDB - Parts 1b, 1c

This picks up from the last post, which you can find here:
http://dansrandombits.blogspot.com/2012/11/python-and-flask-and-mongodb-part-1a.html

This section concerns itself with dealing with MongoDB, then accessing data in Mongo through Flask to display to a web client.

Install Mongo, create database with a collection of three records


First, we have to get Mongo installed with some simple data.   Download MongoDB from http://www.mongodb.org, and use version 2.2 or higher.

Unzip, and move the contents to a new folder--on my Mac, I put the latest MongoDB in my user home directory in its own folder under ~/bin/mongodb, then I symlink it to ~/bin/mongodb/current, and create a data directory as well to hold the Mongo databases. These command lines do the trick after Mongo's downloaded.

mkdir ~/bin/mongodb
tar -zxvf ~/Downloads/mongodb-osx-x86_64-2.2.1.tgz -C ~/bin/mongodb
ln -s mongodb-osx-x86_64-2.2.1/ current
mkdir ~/bin/mongodb/data

Now launch Mongo:
cd ~/bin/mongodb/
./current/bin/mongod -dbpath ./data/db

Add records to Mongo


Open another terminal window, and open the mongo client.  We'll create a database named test, and insert three records into a collection called "fruits"
cd ~/bin/mongodb
./bin/mongo

This gets you a mongo prompt.
show dbs
use test
db.fruits.insert({name: "apple"})
db.fruits.findOne()
db.fruits.insert({name: "orange"})
db.fruits.insert({name: "banana"})
db.fruits.distinct("name")
db.fruits.distinct("name").sort()

Now let's add another field called displayOrder:
db.fruits.update({name: "apple"}, {$set: {displayOrder: 1}})
db.fruits.update({name: "banana"}, {$set: {displayOrder: 2}})
db.fruits.update({name: "orange"}, {$set: {displayOrder: 3}})

Let's find these:
db.fruits.find()
{ "_id" : ObjectId("502f1f4098075d85dc674fb5"), "displayOrder" : 1, "name" : "apple" }
{ "_id" : ObjectId("502f284f98075d85dc674fb7"), "displayOrder" : 2, "name" : "banana" }
{ "_id" : ObjectId("502f284b98075d85dc674fb6"), "displayOrder" : 3, "name" : "orange" }

Connect Flask to MongoDB


This is the last part of this document -- and to do this, we'll add what Flask calls a route.  It'll be our first route, so we'll just call it mongo1.

First, install the pymongo driver.   There's no ORM or anything here; Flask will connect directly to MongoDB.

sudo pip install pymongo

Next, update that hello.py file.  Insert this code below the line that says  return "Hello World!", and before "if __name__ = 'main'"

from pymongo import Connection
conn = Connection('127.0.0.1', 27017)

@app.route("/mongo1")
def find():
    dbTest = conn['test']
    dbDocs = []

    for fruit in dbTest['fruits'].find():
        dbDocs.append(fruit)

    return str(dbDocs)

Stop Flask on the command line if it's still running with a control-C, then relaunch it with python hello.py.  (If you're using PyCharm select the hello.py file in the left-hand Project window, right click, and choose Debug.)

In the mongod window you'll see something like this that lets you know you've connected to Mongo:

Fri Aug 17 13:13:13 [initandlisten] connection accepted from 127.0.0.1:53824 #2 (2 connections now open)

Now in a web browser visit that route at http://127.0.0.1:5000/mongo1

You'll see those three records returned in JSON format, which is effectively how both Python and MongoDB store their data--effectively means it's not quite true, but close enough.

What's happening here?


First of all, the route /mongo1 is bound to the function find(), which makes a connection to the Mongo database ['test'].
dbDocs is declared as an empty list, and then a find request is made to the collection 'fruits'.   The for loop iterates with the results from the find(), and builds up a Python list named dbDocs.  Finally, dbDocs is converted into a string, and returned to the user's web browser, where it appears as unformatted JSON.

That's it for now, but it does show you how you can quickly build a simple web service using Flask and Mongo.

Additional directions to go from here would be to use templates for rendering, getting a bit of styling in place, and isolating the Flask application into its own Python environment.

Python, Flask, MongoDB - Part 1a

The past few months I've been working quite a bit with Flask and MongoDB... and I've found it easy to use and quickly setup a database driven website.   Most of the tutorials and such out there assume some sort of knowledge or experience with Python or Javascript or both--and leave out quite a few details or are just plain outdated.
If you're not familiar with Flask, well... Python Flask is a web micro-framework designed to make creating web services and sites easy and even fun.  You can read all about that here:  http://flask.pocoo.org/ and you can skim through its Quickstart to get a feel for what Flask is about.
Now I'm sure what I'm writing will be outdated in a few months, but as of now, in early November 2012, it's current.  And with enough work collegues and geek friends asking me "how did you get going with Flask and/ or Python and/or Mongo" I might as well share these steps.  So let's go.

Goals for today

These should be enough to get going.
1a. Install Flask and write a Hello World.
1b. Install MongoDB and add a database with a single collection with three records
1c. Get Flask to connect to MongoDB and display the three records
A word about environments: I'll be using a Mac for this, and these steps should work with anything later than MacOS 10.6.  Many of the examples below were done in a terminal window on Mac OS, so you'll need to adjust the commands if you're using another platform.

Environments and prep work

A word about environments: I'll be using a Mac for this, and these steps should work with anything later than MacOS 10.6.  Many of the examples below were done in a terminal window on Mac OS, so you'll need to adjust the commands if you're using another platform.
To develop, I use a bit of vim and a bit more of PyCharm as the IDE.  More on PyCharm in a later post, but if you'd like to download a 30-day trial version you can do that here: http://www.jetbrains.com/pycharm/
Other things that aren't covered in this particular post will be virtual environments, Flask templates, or styling--it's more important to do something than do everything all at once.  Also, there won't be any manipulation of the Mongo data, such as creating, updating, and deleting records.  Those would both be additional posts.
For this first post, it's just straight up Python with globally install Python packages and the like.

Step 1 - Install Flask and write Hello World

Figure out your Python version

Let's Start by figuring out what version of Python you have.  These steps work with Python 2.6 or higher.  At a command line (open up a Terminal window--I use iTerm2 instead) type the following to make sure you've got Python installed:
python --version
That should return something like Python 2.6.6 or Python 2.7.3.  Both are fine.
If you don't have Python 2.6 or 2.7, install it.  Don't install Python 3 for this tutorial.

Install flask and make the first app

Install flask You might have to sudo these.
easy_install pip
pip install flask
Create the flask directory and application
mkdir ~/Desktop/flask
vim ~/Desktop/flask/hello.py
Add these lines:
from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == "__main__":
    app.run()
You can also just create the hello.py using a text editor, but it's important to note that in Python the indents are significant--the return and app.run() statements should be indented four spaces.  This will make copying and pasting code difficult if you're not aware of it.
Finally, run it:
cd ~/Desktop/flask
python hello.py
 * Running on http://127.0.0.1:5000/
Now, in a web browser, go to http://127.0.0.1:5000/
And you'll see Hello World.

Sunday, June 24, 2012

Bootstrapping custom Windows EC2 instances from stock AMIs

Lately I've been playing around with building out EC2 Windows and Linux instances from just plain stock AMIs.    It's no fun managing a separate custom AMI for each of the seven EC2 regions, and when you have two or three custom AMIs to build it's tedious and there are really better ways to spend our time.  After all, it's a pain to keep AMIs up to date, and that goes double for Windows AMIs

Windows AMIs have always been a headache though, with packaging and sysprep, and a lot of it is just tedious voodoo that I feel I shouldn't be wasting my time on.

What I'd like to do is just start with the Amazon-supplied AMIs and bootstrap a full Windows server with all my bits installed into it--and do that from issuing a single command line.   After a bit of tinkering with it last week, I finally got it going, and realized that yep, it can be done, and can be done reliably.


The Goal

A little while ago (April 2012 or so), AWS added the capability to supply userdata to a Windows instance at creation time (more here: https://forums.aws.amazon.com/thread.jspa?messageID=340466)   This is wonderful; Linux distros have had this for a long while, and it makes creating  the exact same system real easy.   Right now, I have some Puppet stuff that can do that for Linux--and Puppet supports Windows now too--so why not try and piggy back on the bit of Puppet infrastructure we have now?

I just want to be automatically create a new Windows EC2 instance, download the most current copy of Puppet installer to it, and then run the installer.  At that point Puppet can manage installation and provisioning of everything else.

The Secret Sauce

Looking around, there have been a few forum postings about attempting to use the new userdata capability towards Windows; but they've mostly been orientated towards getting PowerShell commandlets up and running.  Pass in a <powershell>[script data]</powershell> in userdata and it'll get run!  Sounds exciting.  Well... perhaps... but I've not been a fan of PowerShell: it's syntactically odd, it doesn't follow other familiar language patterns, and it has restrictive execution policies.  I never know if a particular PowerShell script will work on a given box, and to me it's just overcomplicated.

But there's another trick: AWS also allows you to pass in just plain old DOS batch commands which get executed as the local administrator.  That sounds a lot more promising. Pass in userdata like <script>ipconfig > C:\ipconfig-boot.txt</script> and it gets run.    No need to learn PowerShell for the nth time.  So maybe I can just grab our Puppet installer package from some network location and run it!

There's one big drawback though: the DOS command line doesn't have a way to download stuff from a regular HTTP location via curl or wget.   That's a small hurdle, as there's an olded technology that can overcome that: VBScript.   VBScript can do quite a bit, and you can easily script downloads in VBScript by creating a good old XMLHTTP object.  And as an added bonus, that XMLHTTP support exists on all versions of Windows back to XP, and probably back quite a bit more.   VBScripts just run.  It's the way to go.

Now the attack plan is clear:

- Via the command line create a new Windows EC2 instance, with userdata to send some DOS commands
- The new instance runs DOS commands in a Windows batch file.  This batch file writes the VBScript file.
- The last DOS command executes the newly written VBScript file
- The VBScript downloads the installer via HTTP and executes it

The result of last step, when the installer's all finished, has the effect of adding the Windows instance to our managed infrastructure.  At that point Puppet can take over the system building. 



The Script

All of this needs to be put into a script, and surrounded by <script> tags too for EC2 to see it as a script.    Here's what it looks like.

It's nothing too fancy, but it gets the job done--it starts up, connects to an HTTP server, downloads the Puppet Enterprise installer (pe.msi), and then executes it silently, pointing the Puppet agent on this new Windows instance to our Puppetmaster server.

Save this script into a file and call it win-userdata.txt. That will get used from the command line.



The Command Line


Now let's pass that userdata script when we launch a new instance from the command line.

On a machine that has the EC2 API tools installed and configured, issue a command like this:


ec2-run-instances ami-e7ce94a2 -n 1 -g default -k <your-key-name> -t m1.small -z us-west-1a -f win-userdata.txt

That AMI referenced above is the Windows 2008 default AMI for the US-West-1 region in Northern California.

After the command line is executed, you'll see the Windows EC2 instance being spun up.  The installer will be executed very quickly after boot--no need to wait 5 to 15 minutes for the Administrator password.  After about four minutes the new instance will check in with the Puppetmaster for certificate signing (which also can be automated and secured, since you know a lot of info already about the instance you created.)

Now you don't have to download that Puppet installer--that installer that's downloaded can be anything you want--but I find that Puppet makes it easier to keep things tidy and orderly and known.

With this technique, new Windows EC2 instances can be brought online and customized without any more manual intervention than issuing a single command line .  That's a huge win for speeding up deployments and testing, and for getting reliable and repeatable automation going.