mrjob 0.4

Run MapReduce jobs on Hadoop or Amazon Web Services

  Add it to your Download Basket!

 Add it to your Watch List!

0/5

Rate it!
send us
an update
LICENSE TYPE:
The Apache License 2.0 
USER RATING:
UNRATED
  0.0/5
DEVELOPED BY:
David Marin
HOMEPAGE:
github.com
CATEGORY:
ROOT \ System \ Clustering and Distributed Networks
mrjob is a Python module that helps you write and run Hadoop Streaming jobs.

mrjob fully supports Amazon's Elastic MapReduce (EMR) service, which allows you to buy time on a Hadoop cluster on an hourly basis. It also works with your own Hadoop cluster.

Installation:

python setup.py install

Setting up EMR on Amazon

 * create an Amazon Web Services account: http://aws.amazon.com/
 * sign up for Elastic MapReduce: http://aws.amazon.com/elasticmapreduce/
 * Get your access and secret keys (go to http://aws.amazon.com/account/ and click on "Security Credentials") and set the environment variables $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY accordingly

Try it out!


# locally
python mrjob/examples/mr_word_freq_count.py README.md > counts
# on EMR
python mrjob/examples/mr_word_freq_count.py README.md -r emr > counts
# on your Hadoop cluster
python mrjob/examples/mr_word_freq_count.py README.md -r hadoop > counts

Advanced Configuration

To run in other AWS regions, upload your source tree, run make, and use other advanced mrjob features, you'll need to set up mrjob.conf. mrjob looks for its conf file in:

 * ~/.mrjob
 * mrjob.conf anywhere in your $PYTHONPATH
 * /etc/mrjob.conf

See mrjob.conf.example for more information.

Last updated on August 31st, 2012

feature list requirements

#MapReduce framework #Python framework #Hadoop Streaming #MapReduce #Python #framework #library

Add your review!

SUBMIT