Using AWS CLI to connect to MySQL on RDS

I messing around with RDS on Amazon Web Service for a project im working on and realized that there is not a true way to connect to the instance that the database is running on when you stand up a MySQL instance.

After poking around a little bit I came up with this solution and it worked ok for me. Probably needs to be refined and there is probably an easier way also but this worked for me.

First im going to assume you have a MySQL instance running in RDS. Once you have this done you will need to download and unzip the AWS CLI tool to a directory you have access to. > http://aws.amazon.com/developertools/2928

Once you have done this you will need to unzip the package. You can check out the README if you want or you can just do this…
1. copy the credential-file-path.template file and call it something like “cred-file” then chmod it 600
2. vi cred-file fill in the AWSAccessKeyId and AWSSecretKey values. You can get these values from IAM control panel in AWS under your user. If you are lost on that there’s a handy tool called Google. ūüôā
3. You will need to set AWS_RDS_HOME in your path
4. Make sure JAVA_HOME is set in your path

Once you have followed these steps you can now try it out…
Do: $AWS_RDS_HOME/bin/rds-describe-db-instances --aws-credential-file cred-file --headers
Hopefully your formatting will be better than what you see here but you should see something similar to this:

DBINSTANCE DBInstanceId Created Class Engine Storage Master Username Status Endpoint Address Port AZ SecAZ Backup Retention Multi-AZ Version License Publicly Accessible Storage Type
DBINSTANCE blah 2014-10-15T17:36:01.553Z db.t1.micro mysql 5 blah available blah.czbrj4wkmqs3.us-east-1.rds.amazonaws.com 3306 us-east-1b us-east-1a 7 y 5.6.19 general-public-license y gp2
SECGROUP Name Status
SECGROUP default active
PARAMGRP Group Name Apply Status
PARAMGRP default.mysql5.6 in-sync
OPTIONGROUP Name Status
OPTIONGROUP default:mysql-5-6 in-sync

Now you have tested and know you can access your MySQL db in RDS.

Advertisements

Mount a shared drive in virtualbox

This is mainly because I keep forgetting the commands for this…

sharename="whatever.you.want.to.call.it"; 
sudo mkdir /mnt/$sharename
sudo chmod 777 /mnt/$sharename
sudo mount -t vboxsf -o uid=1000,gid=1000 $sharename /mnt/$sharename
ln -s /mnt/$sharename $HOME/Desktop/$sharename

 

Then add this line to your /etc/rc.local file so that it auto-mounts when you start up your vm.

Setting up Open Chef Server and a Workstation

The quick and dirty way…

Prep work

For me, I stood up two Centos 6.5 instances in virtualbox. I did the basics of installing the EPEL repo, groupinstall of “Base” and “developer tools” in YUM, setup and configured a user for myself.

Create my user:

useradd -m -G wheel seth
passwd seth

Install EPEL repo:

wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
sudo rpm -Uvh remi-release-6*.rpm epel-release-6*.rpm

YUM Groupinstall:

sudo yum groupinstall base "developer tools"


Downloading the needed files

http://www.getchef.com/chef/install/

First we will install the server…

 

For the server

Pick which package you need here and download it to the “chef-server”

cd /opt
sudo wget "https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-server-11.0.12-1.el6.x86_64.rpm"
sudo rpm -ivh  chef-server-11.0.12-1.el6.x86_64.rpm
sudo chef-server-ctl reconfigure

This command will set up all of the required components, including Erchef, RabbitMQ, and PostgreSQL

 sudo chef-server-ctl test

 

For the Workstation

As root run the curl command supplied to you from the Chef Download page. For me it was…

curl -L https://www.opscode.com/chef/install.sh | bash

Once this completes do this to test the install and see the version number:

chef-client -v

Now install GIT…

yum install git

(This will install version 1.7.1-ish)

From here as my own user I went to my home directory and cloned down the Chef GIT repo:

git clone git://github.com/opscode/chef-repo.git

Next cd into the chef-repo dir and create your .chef folder. This folder will hold your authorization files in a few steps from now.

cd chef-repo
 mkdir .chef

Add your .chef to the .gitignore file so it and its contents will not be uploaded.

echo .chef >> .gitignore

Get the .pem files and knife.rb files
You will need to copy these files from the chefserver that you previously setup. These files essentially give you access to connect to the server via chef-client and with knife. For me the command run from my workstation was…

sudo scp root@192.168.1.152:/etc/chef-server/admin.pem /home/seth/chef-repo/.chef/admin.pem
sudo scp root@192.168.1.152:/etc/chef-server/chef-validator.pem /home/seth/chef-repo/.chef/chef-validator.pem

You then want to run the knife initial command to create the knife.rb file in your .chef folder. Unless you know the hostname of your server and the locations of your admin and chef-validator.pem files this step will fail. You also need to make sure that port 443 is open on your server. You can do that with this:

sudo iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
sudo service iptables save

Now do this:

knife configure --initial

Here is what my output looked like to give you an idea of what to expect:

[seth@chefworkstation001 .chef]$ knife configure --initial
 Overwrite /home/seth/.chef/knife.rb? (Y/N)y
 Please enter the chef server URL: [https://chefworkstation001.devopshomelab:443] https://192.168.1.152:443
 Please enter a name for the new user: [seth]
 Please enter the existing admin name: [admin]
 Please enter the location of the existing admin's private key: [/etc/chef-server/admin.pem] /home/seth/chef-repo/.chef/admin.pem
 Please enter the validation clientname: [chef-validator]
 Please enter the location of the validation key: [/etc/chef-server/chef-validator.pem] /home/seth/chef-repo/.chef/chef-validator.pem
 Please enter the path to a chef repository (or leave blank):
 Creating initial API user...
 Please enter a password for the new user:
 Created user[seth]
 Configuration file written to /home/seth/.chef/knife.rb

Once you successfully make it past this point you should be good to go. You can test your configuration and connectivity to your server by running the following two commands:

knife client list
knife user list

Now lets bootstrap our new chef workstation

On the server you will need to configure knife like we did on the client side so do:

sudo knife configure -i

You will see something similar to what we saw the firwst time we ran this. Here is my output…

WARNING: No knife configuration file found
Where should I put the config file? [/root/.chef/knife.rb]
Please enter the chef server URL: [https://chefserver.devopshomelab:443]
Please enter a name for the new user: [seth] chefserver
Please enter the existing admin name: [admin]
Please enter the location of the existing admin's private key: [/etc/chef-server/admin.pem]
Please enter the validation clientname: [chef-validator]
Please enter the location of the validation key: [/etc/chef-server/chef-validator.pem]
Please enter the path to a chef repository (or leave blank):
Creating initial API user...
Please enter a password for the new user:
Created user[chefserver]
Configuration file written to /root/.chef/knife.rb

If you have everything configured correctly you can now bootstrap your workstation by running the following command:

sudo ./knife bootstrap 192.168.1.153 --sudo -x seth -N "chefworkstation001"

 

You should have successfully installed and configured a working Chef open server and workstation following these steps. If you have questions or comments please feel free to use the comment section below. As always you can also consult the official Chef documentation on this subject by going here:

SERVER: http://docs.opscode.com/install_server.html

WORKSTATION: http://docs.opscode.com/install_workstation.html

Also check out my other post about bootstrapping a node in your new Chef environment. HERE

Getting setup with MongoDB and Python, Pymongo, and Bottle on Centos 6

From my older blog. Originally posted Oct 26th 2012

I had to walk through all of this to get setup for my Mongo Developer and Administrator certification classes. I figured I might help you out and put it all together instead of you having to hunt all over the place for this info.

Lets get started…

Step 1: Install MongoDB

To install MongoDB on your fedora system, issue the following commands in your terminal:

su -

Lets make sure that mongo hasnt already been installed while you were playing around with it…

yum -y remove libmongodb mongodb mongodb-server

Setup the mongodb repo…

vi /etc/yum.repos.d/10gen.repo

For 32 Bit, place the following code on that file:
http://gist.github.com/3949816#file_32bit.repo

and for 64 Bit this one:
http://gist.github.com/3949816#file_64bit.repo

Installation…
Run these two commands.

yum install mongo-10gen mongo-10gen-server
mongod -f /etc/mongod.conf

Then open your Mongo Shell in terminal (to ensure it’s functional) with:

mongo

and close it by pressing Ctrl+C; Then add the mongo service to the system’s startup with:

chkconfig --levels 235 mongod on

Reboot and reopen the Mongo Shell with the command:

mongod -f /etc/mongod.conf && mongo

Close it with Ctrl+C and then go to:
http://localhost:27017/
In your web browser, you should see something like this:

“You are trying to access MongoDB on the native driver port. For http diagnostic access, add 1000 to the port number”

If you see this then you have it installed and setup correctly.

Step2. Installing Python, Pymongo, and Python-Bottle

First a little background…

Python – Python is a general-purpose, interpreted high-level programming language whose design philosophy emphasizes code readability. Its syntax is said to be clear and expressive.Python has a large and comprehensive standard library. More info can be found here: http://www.python.org/

Pymongo – PyMongo is a Python distribution containing tools for working with MongoDB, and is the recommended way to work with MongoDB from Python. More info can be found here: http://api.mongodb.org/python/current/

Bottle – Bottle is a fast, simple and lightweight WSGI micro web-framework for Python. It is distributed as a single file module and has no dependencies other than the Python Standard Library. More info is here: http://bottlepy.org/docs/dev/

If you¬†haven’t¬†already you need to install the EPEL repo for Centos 6

Check to see if you already have it installed or not…

yum repolist

You will see something similar to this:

repo id      repo name              status
base         CentOS-6 - Base        6,294
extras       CentOS-6 - Extras      4
updates      CentOS-6 - Updates     830
repolist:                           7,128

Since you dont see it above you need to grab the package…
http://mirror.us.leaseweb.net/epel/6/i386/repoview/epel-release.html

-OR-

wget "http://mirror.us.leaseweb.net/epel/6/i386/epel-release-6-7.noarch.rpm"

Note… the version number in the rpm name…if you are looking at this very late after the pub date of this post then you likely need to grab a more current version. Just follow the link above and you should see it.

do this next to test the rpm…

rpm -ivh epel-release-6-5.noarch.rpm --test

You will get back a NO KEY message.

You now need to download and install the key from FedoraProject and install on your system…

wget https://fedoraproject.org/static/0608B895.txt
mv 0608B895.txt /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6

Verify that the key got installed successfully…

rpm -qa gpg*
gpg-pubkey-0608b895-4bd22942

Now install the epel-release-6-6.noarch package, which will enable EPEL repository on your system…

rpm -ivh epel-release-6-5.noarch.rpm

Now verify that the EPEL repo is working correctly…

yum repolist

You should see a line that looks similar to this:

epel      Extra Packages for Enterprise Linux 6 - x86_64      7,345

Now lets get Python, Pymongo and Bottle setup…

Pretty simple and straight forward…

sudo yum -y install python python-setuptools python-setuptools-devel
sudo yum -y install python-bson pymongo pymongo-gridfs

To install bottle you may need to install PIP first…
You can do this using curl and python…

curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py | python

Then install bottle…

pip install bottle

Allow myself to introduce…myself

Hey im Seth Floyd. Im starting this blog as both a place to store notes and to give an example and display of what im capable of in the world of system administration and DevOps.

Image

A little background first…
I went to school to be a chef and restaurant manager. I never took a computer class my entire time there. One roommate I had was a Computer Science major and he had this computer in our apartment that he used for writing papers. he was older than me and I knew he would be leaving soon so I needed to get my own computer for writing my own papers so I didn’t have to live in the library. Once I talked my parents into getting me one I started playing around with it. The OS was Windows 3.1 and it was fully loaded with AOL and Word Perfect.
Step forward a few years and I had finished school and purchased my own computer. I played games and messed around with different graphics programs and figured out how to upgrade to Windows 95 and then to Windows 98 all on my own. From here I progressed when I got a copy of Windows NT and shortly after I took my first IT job. I worked for a company that handled the software for hotels and this included all of the accounting and reservations and other misc software needs like food and beverage. this was my first taste of UNIX….SCO Unix. This was 1999.

At this job I was clueless and my boss let me know that every day. he set my .profile to read, “When Seth gets a clue he will know how to change this to say something else.”. I saw that every single time I logged in and it finally started bugging me. he tried to give me an another guy a crash course in UNIX but we had no clue what he was talking about. I actually didn’t grasp the concept that there were other OS’s out there and UNIX was one of them. “Cat” blew my mind.

Flash forward another year and I was working at Mindspring as a dial-up tech and heard some guys talking about this thing called Redhat Linux. I figured it might be something I needed to know so I got a O’reilly book called “Redhat Linux Administration”. I clearly remember reading through one chapter and the light came on…”That’s what I was doing at that job before!”. Once I realized that then the pieces really started falling together.

I stayed there a few more years and didn’t really have any idea of what I wanted to do. About that time my department was laid off so I took a break from IT for a few years but still dabled at home with different linux distros. About 5 years and 2 kids later I again took an IT job with a consulting company who specialized in Apple computers. I got ramped up pretty quickly by reading a manual and pouring through some of the more popular Mac forums. I already was familiar with Free BSD from my days at Mindspring so it was interesting to me to have a OS that was easy to move around in and also be able to administer using linux commands and tools.

After a brief stint here I joined up with another company. I started out in tech support and worked my way up to a Sr tech support position and stayed there for 3 years. During that time I worked on all things database related. We are an Oracle shop and I was sure that was what I wanted to do. I even went as far as getting my certification in Oracle database as a OCA.

I eventually moved to the configuration management team. the intent by the director of that department was for me to learn the back-end environments and then transition over to the non-production dba team. That never happened. The director was let go, his role was eliminated and the non-prod dba team was rolled into the production team. Through a turn of events I was basically shut out of any chance of a dba role – junior or otherwise. We silently declared that we will only hire from outside the company for DBA roles and they will all be for Sr DBAs as well.

Dream. Crushed.

But at that same time there was a flicker of a flame of another fire starting to grow a little more and more. I was getting the hang of this CM gig. I was setting up jobs in Jenkins, writing little scripts to automate things here and there using Bash, feeling more comfortable with the linux OS – moving file around, installing and maintaining applications, and even administering some less important servers and systems. Some of my other duties were/are getting up at 4A.M. to make sure our nightly deployments to trunk have completed successfully, administering our Confluence instance, and writing a whole lot of documentation.

We eventually started using this thing called “Chef” in our deployments. I kept hearing about “databags” and “nodes” and “recipes” and of course got curious. I started reading up on this and trying to learn it using learnchef.opscode.com. I had so much trouble getting everything installed correctly I kind of just put it to the side and decided I wasnt ready for that yet. In looking for more info on Chef I kept coming across this other thing called “Puppet”. Its a similar tool to Chef and can be used for the same thing. We dont use it here at the office but I wished we did. It turns out its a lot easier to learn and seems to make more sense to me. It also seems to have a bigger and more active community so its easier to find tutorials and help with things.

So this all brings me to the reason and purpose of this blog…

I want to “do” DevOps. I tend to lean more toward the Ops part of this world but thats mainly because I hate coding. My attention span is too short to just sit down and type out code for something. Part of my frustration with coding comes from the lack of properly learning the concepts of programming and stuff that I would have gotten from classes in college. I have started working on getting my RHCE and I think after that I will pursue a certification in Puppet. This blog will be a place to store my notes, ask for help, help others, and to demonstrate what I can do even though I might not have “professional working experience” in certain subjects.

Stay tuned…this might be interesting…or a train wreck.

Seth