Knife EC2 Server Create Error: Authentication failed

Sometimes with all of the rush and trying to keep track of a 1000 moving parts you might get stumped by a fairly simple issue. Here are a few things to check if you get hung up with an “Authentication failed for user” error when running a “Knife ec2 server create” command.

Waiting for sshd access to become availabledone
Connecting to 52.5.159.42
Failed to authenticate ec2-user - trying password auth
Enter your password:
ERROR: Net::SSH::AuthenticationFailed: Authentication failed for user ec2-user@52.5.159.42@52.5.159.42

Do you have your .pem file downloaded and installed with the correct permissions on your workstation you are running the command from?

It should be in the EC2-USER’s .ssh dir -> /home/ec2-user/.ssh

Make sure its chmodded 400

Make sure you have the knife.rb file set correctly to reference the .pem file for you…otherwise you will have a lot of typing for your command.

knife[:identity_file] = "/home/ec2-user/.ssh/aws-seth.pem"

Make sure you are using the correct user. Unless you have specifically changed something in your configurations by default you will be connecting as the “ec2-user”. So make sure thats what is trying to connect in your error output.

Hopefully these tips will help you narrow down the issue. You have to think about whats really happening and from where with Chef some sometimes these simple issues can really drive you nuts.

Setting up a Chef workstation with ChefDK

*This is assuming you are running on CentOS or some other RHEL platform

Download the Chef-DK package…
Go to: http://downloads.getchef.com/chef-dk/
Install the package…

sudo rpm -Uvh ChefDK.....rpm

Once its installed check it and make sure the install was successful…
Do:

sudo chef verify

 


Set System Ruby

Do:

which ruby

You might see something like this: ~/.rvm/rubies/ruby-2.1.1/bin/ruby
If you want to use the version of ruby that came with ChefDK do the following…assuming you are using BASH…
Do:

echo 'eval "$(chef shell-init bash)"' >> ~/.bash_profile

Do:

source ~/.bash_profile

Do:

which ruby

Install Git if you dont already have it…

sudo yum git install

 


Setting up the chef-repo

You can do this two ways….download the starter kit from your Chef server OR manually. In this case we will do this manually because I already happen to have a hosted Chef account and am also using my keys on other instances and dont want to have to set them all up again. So…go to your home directory and do:

git clone git://github.com/opscode/chef-repo.git

Then go to ~/chef-repo/ and do:

mkdir .chef

Three files will need to be placed in this directory:
– knife.rb
– ORGANIZATION-validator.pem
– USER.pem

In order to not upload your .chef directory which will house your certs do this:

echo '.chef' >> ~/chef-repo/.gitignore

Now you need to get the 3 files that go into your .chef directory.
Log onto your Chef server. For me this is located at: https://manage.opscode.com

Once logged in click ADMINISTRATION at the top then the name of your organization.

Knife.rb – Click “Generate Knife Config” and download the file. Place it in your .chef directory
ORGANIZATION-validator.pem – can be downloaded by clicking “Reset Validation Key” in the Administration page.
USER.pem – can be downloaded by clicking Users on the left hand side and then choosing your username, and finally clicking “Reset Key

 



Add Ruby to your Path

DO:

echo 'export PATH="/opt/chefdk/embedded/bin:$PATH"' >> ~/.configuration_file && source ~/.configuration_file

Now lets verify that we are all set…
DO:

cd ~/chef-repo

DO:

knife client list

You should see a list of your clients which will only be the one you are on for right now.

That’s it. Let me know if you have questions or run into issues or see mistakes.

AWS Loadbalancing

For previous setup info see my post on setting up instances using Chef and AWS: HERE

AWS Loadbalancing

If you have completed the steps in the previous post you can repeat them a few times to spin up 3 or more instances. For this post lets assume I have spun up 3 instances named web001, web002, and web003.

1. In your AWS control panel navigate to your EC2 instances and on the left hand column click “Load Balancers” under “Network and Security”.

2. Click the “Create Load Balancer” button at the top.

3. Define Load Balancer Enter a name for your LB and for this example use “EC2-Classic”. By default port 80 is being monitored and since we have 3 web servers running httpd this will be all we need. Click Continue.

4. Configure Health Check For this example the only thing that you need to change is PING PATH. Change it to just “/” dropping the index.html that is already there. Leave everything else the way it is. Continue

5. Add Instances to Load Balancer Check off your 3 instances. Leave the settings for Availability Zone Dist as they are. Continue

6. Keys Here you can create a key and value pair for your load balancer. For example, you could define a tag with key = Name and value = Webserver. Continue

7. Review This will give you a run down of your load balancer before you create it. Click CREATE and it will be started up.

Once your LB has been started it will take a couple of minutes before your instances are in service. If you click the name of your LB  you can see the description tab below. Wait a couple of minutes and then copy the DNS Name which should be the first line that ends with (A Record) Ex. LB-001-201566711211.us-east-1.elb.amazonaws.com Paste this into your browser and you should see your webpage that you created in your instances.

Tabs

You have 7 tabs that contain information about your loadbalancer.

1. Description DNS Name can be found here along with brief info on the status fo the instances connected to this LB

2. Instances Status of the instances that are connected to the LB. You can also drop the instance from the LB here

3. Health Check The main thing to watch here is “Unhealty Threshhold”  You can also edit the Health Check settings here.

4. Monitoring CloudWatch metrics for the selected resources. You can also create alarms for your instances on this page.

5. Security Displays your security groups that are connected to this LB

6. Listeners The ports and protocols you are listening for on your instances.

7. Tags Tags that you created when setting up the LB and you can also create more from this page as well

 

This is a VERY basic run down of how to create a load balancer in Amazon AWS. If you have any questions or input feel free to use the comment section below.

 

Chef: Spin-up and bootstrap a node on AWS

A few quick instructions for spinning up a node on AWS using Chef and the Knife EC2 plugin.
Some of these steps I culled together from various sites and thought it might be helpful to have them all in one spot.

Base box: This is running from an instance I stood up on AWS (redhat)

Prep work:

DO: yum groupinstall Development tools
DO: Install Chef Client: curl -L https://www.opscode.com/chef/install.sh | bash
DO: Download your starter kit from your Chef Hosted account and unzip in /opt/chef/ (assuming you have a hosted account already setup there)
DO: copy your AWS key file (.pem) to your /home/ec2-user/.ssh/ folder so your workstation can connect back to AWS to spin up new instances. (There may be a different or better way of doing this but it worked for me)

 

Install the Knife EC2 plugin…
sudo /opt/chef/embedded/bin/gem install knife-ec2

Configure the knife.rb for AWS…(knife.rb is located at something like… /opt/chef/chef-repo/.chef/knife.rb)

Knife File
____________________________________________________________________________________________

current_dir = File.dirname(__FILE__)
log_level :info
log_location STDOUT
node_name "YOUR_USERNAME"
client_key "#{current_dir}/YOUR_USERNAME.pem"
validation_client_name "YOUR_ORGNAME-validator"
validation_key "#{current_dir}/YOUR_ORGNAME-validator.pem"
chef_server_url "https://api.opscode.com/organizations/YOUR_ORGNAME"
cache_type 'BasicFile'
cache_options( :path => "#{ENV['HOME']}/.chef/checksums" )
cookbook_path ["#{current_dir}/../cookbooks"]

#A note about where to find this info…
#You will need to log into your AWS IAM (https://console.aws.amazon.com/iam) Go to users, click your #username and you should see an area that says “Access Credentials”. You will likely need to create new #ones if you didnt already know what your keys were at this point. Do so and then make sure you download #them and keep in a safe spot. Fill out the next two lines…the rest is optional at this point but #useful to know that you can have this here.

knife[:aws_access_key_id] = “AWS_ACCESS_KEY_ID”
knife[:aws_secret_access_key] = “AWS_SECRET_ACCESS_KEY”

# Default flavor of server (m1.small, c1.medium, etc).
knife[:flavor] = “m1.small”

# Default AMI identifier, e.g. ami-12345678
#knife[:image] = “ami-b06a98d8”

# AWS Region
#knife[:region] = “us-east-1”

# AWS Availability Zone. Must be in the same Region.
#knife[:availability_zone] = “us-east-1b”

# A file with EC2 User Data to provision the instance.
#knife[:aws_user_data] = “”

# AWS SSH Keypair.
##knife[:aws_ssh_key_id] = “Name of the keypair you want to use in AWS…Instance> Keypair”
___________________________________________________________________________________________

 

Final Command

Finally run this command to connect to AWS and start up your instance and install the recipe:
sudo knife ec2 server create -x ec2-user -S *KEY_NAME* -G default,www -r 'recipe[apache2]' -I ami-b06a98d8

-x = user that we’ll connect with SSH
-S = AWS SSH keypair
-G = Security groups
-r = recipes to run once the instance is up
-I = AMI Identifier e.g. ami-12345678

(This is assuming you already know how to store your cookbooks locally and upload them to your Chef server. )

Once this completes you should have a new instance on AWS with a working apache2 instance and should be able to navigate to the webpage you created for Apache. A separate post will be coming soon to cover these steps.

 

Possible Error

You could possibly run into this:
[fog][WARNING] Unable to load the ‘unf’ gem. Your AWS strings may not be properly encoded.
ERROR: Excon::Errors::SocketError: getaddrinfo: Name or service not known (SocketError)

If so install UNF via gem like so:
sudo /opt/chef/embedded/bin/gem install unf
Try your create command again…

Once you are finished playing around you can use Knife EC2 to actually delete the AWS instance and also remove the node from your hosted Chef management Console:
sudo knife ec2 server delete -y –purge INSTANCE_IDENTIFIER
sudo knife ec2 server delete -y --purge i-e639ca0d

 

Chef Error: Knife configure

When doing your initial “knife configure -i” command while setting up a chef workstation, if you encounter this error:

ERROR: Errno::EHOSTUNREACH: No route to host - connect(2)

Make sure you check your firewall settings.

On CentOS you can do:

sudo iptables -S

This will show you what is enabled currently. If you don not have port 443 open you will run into issues. To open it you can do this:

-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT

Make sure you save your changes…

sudo service iptables save

…and restart the firewall…

sudo service iptables restart

Setting up Open Chef Server and a Workstation

The quick and dirty way…

Prep work

For me, I stood up two Centos 6.5 instances in virtualbox. I did the basics of installing the EPEL repo, groupinstall of “Base” and “developer tools” in YUM, setup and configured a user for myself.

Create my user:

useradd -m -G wheel seth
passwd seth

Install EPEL repo:

wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
sudo rpm -Uvh remi-release-6*.rpm epel-release-6*.rpm

YUM Groupinstall:

sudo yum groupinstall base "developer tools"


Downloading the needed files

http://www.getchef.com/chef/install/

First we will install the server…

 

For the server

Pick which package you need here and download it to the “chef-server”

cd /opt
sudo wget "https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-server-11.0.12-1.el6.x86_64.rpm"
sudo rpm -ivh  chef-server-11.0.12-1.el6.x86_64.rpm
sudo chef-server-ctl reconfigure

This command will set up all of the required components, including Erchef, RabbitMQ, and PostgreSQL

 sudo chef-server-ctl test

 

For the Workstation

As root run the curl command supplied to you from the Chef Download page. For me it was…

curl -L https://www.opscode.com/chef/install.sh | bash

Once this completes do this to test the install and see the version number:

chef-client -v

Now install GIT…

yum install git

(This will install version 1.7.1-ish)

From here as my own user I went to my home directory and cloned down the Chef GIT repo:

git clone git://github.com/opscode/chef-repo.git

Next cd into the chef-repo dir and create your .chef folder. This folder will hold your authorization files in a few steps from now.

cd chef-repo
 mkdir .chef

Add your .chef to the .gitignore file so it and its contents will not be uploaded.

echo .chef >> .gitignore

Get the .pem files and knife.rb files
You will need to copy these files from the chefserver that you previously setup. These files essentially give you access to connect to the server via chef-client and with knife. For me the command run from my workstation was…

sudo scp root@192.168.1.152:/etc/chef-server/admin.pem /home/seth/chef-repo/.chef/admin.pem
sudo scp root@192.168.1.152:/etc/chef-server/chef-validator.pem /home/seth/chef-repo/.chef/chef-validator.pem

You then want to run the knife initial command to create the knife.rb file in your .chef folder. Unless you know the hostname of your server and the locations of your admin and chef-validator.pem files this step will fail. You also need to make sure that port 443 is open on your server. You can do that with this:

sudo iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
sudo service iptables save

Now do this:

knife configure --initial

Here is what my output looked like to give you an idea of what to expect:

[seth@chefworkstation001 .chef]$ knife configure --initial
 Overwrite /home/seth/.chef/knife.rb? (Y/N)y
 Please enter the chef server URL: [https://chefworkstation001.devopshomelab:443] https://192.168.1.152:443
 Please enter a name for the new user: [seth]
 Please enter the existing admin name: [admin]
 Please enter the location of the existing admin's private key: [/etc/chef-server/admin.pem] /home/seth/chef-repo/.chef/admin.pem
 Please enter the validation clientname: [chef-validator]
 Please enter the location of the validation key: [/etc/chef-server/chef-validator.pem] /home/seth/chef-repo/.chef/chef-validator.pem
 Please enter the path to a chef repository (or leave blank):
 Creating initial API user...
 Please enter a password for the new user:
 Created user[seth]
 Configuration file written to /home/seth/.chef/knife.rb

Once you successfully make it past this point you should be good to go. You can test your configuration and connectivity to your server by running the following two commands:

knife client list
knife user list

Now lets bootstrap our new chef workstation

On the server you will need to configure knife like we did on the client side so do:

sudo knife configure -i

You will see something similar to what we saw the firwst time we ran this. Here is my output…

WARNING: No knife configuration file found
Where should I put the config file? [/root/.chef/knife.rb]
Please enter the chef server URL: [https://chefserver.devopshomelab:443]
Please enter a name for the new user: [seth] chefserver
Please enter the existing admin name: [admin]
Please enter the location of the existing admin's private key: [/etc/chef-server/admin.pem]
Please enter the validation clientname: [chef-validator]
Please enter the location of the validation key: [/etc/chef-server/chef-validator.pem]
Please enter the path to a chef repository (or leave blank):
Creating initial API user...
Please enter a password for the new user:
Created user[chefserver]
Configuration file written to /root/.chef/knife.rb

If you have everything configured correctly you can now bootstrap your workstation by running the following command:

sudo ./knife bootstrap 192.168.1.153 --sudo -x seth -N "chefworkstation001"

 

You should have successfully installed and configured a working Chef open server and workstation following these steps. If you have questions or comments please feel free to use the comment section below. As always you can also consult the official Chef documentation on this subject by going here:

SERVER: http://docs.opscode.com/install_server.html

WORKSTATION: http://docs.opscode.com/install_workstation.html

Also check out my other post about bootstrapping a node in your new Chef environment. HERE

RUNIT and the ruby syntax error

Ive been working on writing my own cookbook to standup a fully ready to run ELK (Elasticsearch,Logstash,Kibana) server on CentOS. In doing this I have run into a few minor issues of compatibility here and there but nothing major. Once thing that I did find that was particularly troublesome was an error when I was trying to knife up a cookbook for “runit”. Its one of the dependancies for the chef-kibana cookbook and it relys on the “yum-epel” cookbook. I was able to knife up the yum-epel cookbook just fine but when I tried the runit cookbook I hot an error that said:

[seth@localhost cookbooks]$ knife cookbook upload runit yum-epel
Uploading runit [1.5.10]
FATAL: Cookbook file test/spec/libraries/provider_runit_service_spec.rb has a ruby syntax error:
FATAL: /home/seth/chef/chef-repo/.chef/../cookbooks/runit/test/spec/libraries/provider_runit_service_spec.rb:62: syntax error, unexpected '}', expecting tASSOC
FATAL: { provider.load_current_resource }.should raise_error
FATAL:                                                                          ^

My first mistake was installing and using RVM for Ruby management from way before I installed Chef. RVM is too big and robust and handles too many things to try and make it play nice and only worry about managing ruby for me. I uninstalled that and went with RBENV. This is the recommended manager of Ruby from Chef anyways. Once I got rbenv setup and installed I installed ruby 1.9.3p545. I tried again with the knife upload and I still got the same thing. I decided that maybe my version of Chef needed to be updated. I was on 11.2. I reinstalled Chef and tried the knife again. No luck. Same error…thanks for playing, try again.

Now I take to Google to see if this is something unique to me or have I found some bug somewhere. I came across this page and even though it was a year old it did help me out some. https://github.com/rcbops/chef-cookbooks/issues/352 The fix came to me when I re-downloaded the runit cookbook from https://github.com/hw-cookbooks/runit instead of using the cookbook I downloaded directly from Chef Community. For some reason the version was off by one iteration.

[seth@localhost cookbooks]$ knife cookbook upload runit
Uploading runit [1.5.11]
Uploaded 1 cookbook.

Success. Theres not really a better explanation as to what really was wrong here but I do feel it had more to do with the ruby installation than anything else. If you come across this and figure something out please feel free to leave a comment below.