Debug SSH Connection (Password less)

When you are trying set up SSH access to an AWS instance with your private key, sometimes you may run into errors:

Received disconnect from xxx.xxx.xxx.xxx: 2: Too many authentication failures for ubuntu

The error message you get from that is not very helpful generally.

Things you can try is to add ‘-v’ to the SSH command, it will give you more debug information.

However, those debug information is still not enough and misleading.

To get an absolute real problem, you should get into the remote box and read the auth.log file – this will give you exact errors you are having, which makes this kinda of problems much easy to fix.

tail -f /var/log/auth.log

Headless Ubuntu Package Installation

With DevOps concept in mind, I don’t normally set up a server manually anymore. I either use Puppet (Chef, Ansible) or write a quick Bash script to do the server provisioning. What happens is sometimes a package I need to install may pop up a blue GUI asking for user inputs. This is annoying as it could break my server provisioning automation. No worries – I can use debconf-set-selections(Ubuntu) to preconfigure the user inputs.

Let us use an example, today I need to add newrelic-php5 to a server provisioning script written in Bash.

First let me see if I have any preconfigured values in place for newrelic-php5:

# debconf-get-selections |grep newrelic
0

Good, let me go ahead and install it.

# apt-get install newrelic-php5 -y

newrelic_php5_headless_installation

As you can see, it asks me for newrelic license and followed by application name. After filling the details and finishing the installation, I run this command again to see if there is any preconfigured values saved in debconf database:

# debconf-get-selections |grep newrelic
newrelic-php5 newrelic-php5/5.1-deprecation note
newrelic-php5 newrelic-php5/application-name string PHP Application
newrelic-php5 newrelic-php5/license-key string xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

This time I got something. To install this package headless, since I was asked to two inputs during the installation, one for license and one for application name, all I need to do is to preseed these two values with debconf-set-selections.

Let me start it over.

First, remove the package and clean up the debconf:
# apt-get remove newrelic-php5 -y
# echo PURGE | debconf-communicate newrelic-php5

OK, let me preseed the values:
# debconf-set-selections <<< "newrelic-php5 newrelic-php5/application-name string myappname" # debconf-set-selections <<< "newrelic-php5 newrelic-php5/license-key string xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

Verify they are being set:
# debconf-get-selections |grep newrelic
newrelic-php5 newrelic-php5/application-name string myappname
newrelic-php5 newrelic-php5/license-key string xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Try install it again:

# apt-get install newrelic-php5 -y

Boom. It works. Happy scripting.

Centos 6.2 Net Install vs Minimal Install

The other day, I was asked to set up Zabbix monitor system for a client. Quickly checked up, it runs well in Centos – my favour Linux. Let us get started.

I always prefer to use latest version of Centos, as the date of this article, Centos 6.2 just came out.

To avoid big download bandwidth to the client’s office, I was looking two options: 1, Centos 6.2 Minimal  2, Centos 6.2 Net Install

You can find both versions for download in here: (ISO)

http://isoredirect.centos.org/centos/6/isos/i386/

Centos 6.2 Minimal Installation

The ISO only 200+MB. With this ISO, you can install a very very basic running Centos system, not even the base tools. You are mostly like need to ‘yum groupinstall base’ after system boots up. I would go this option, but I don’t want to waste too much time wondering which packages are missing and reinstalling them.

PS. During the installation, you won’t be asked to choose packages to install. You can only install more packages with Yum after.

Centos 6.2 Net Installation

ISO is 100+MB. You will need network for sure. During installation, it will ask you to config the network (or just DHCP). You will also need to put in the Centos 6.2 online images: [Your nearest Centos image mirror]/centos/6.2/os/i386/

The installation took me a while and you will be asked to choose couple of installation types: Desktop, Minimal, Basic Server …. For the difference, please see this link: CentOS 6 “Default” Installation Options

I chose “Basic Server”. After installation, the whole system is about 1.5G.

Hope this would help some one with the Centos 6 installation.

Set up Zend Server and Magento on Centos 6

Finally got a chance to write up a quick guide on how to set up Magento running on Zend Server Community on Centos 6.

Server Hosting

I have been wondering around between Rackspace and Ninefold. My website is gonna be for Australian market only. Ninefold is clearly faster choice, but more expensive, especially when I don’t know whether my website can make money or not. I have been using Rackspace for a while and really enjoy their service. The only thing I am worry about is the SEO impact if I host an Australian website on U.S. After days of researching, I decided to give Rackspace a go, and I think if the website goes well, I may just move the website back to Australia eventually.

Install Zend Server

In here, I have to mention on Magento site, there is a bundle install script for Zend + Magento, however, I tried that and the script doesn’t work anymore. They should have updated that script more frequently. Anyway, let us do our own.

Change SELinux setting

setenforce permissive

Add Yum repos for Zend

vim /etc/yum.repos.d/zend.repo

Paste the content below into zend.repo

[Zend]
name=Zend Server
baseurl=http://repos.zend.com/zend-server/rpm/$basearch
enabled=1
gpgcheck=1
gpgkey=http://repos.zend.com/zend.key

[Zend_noarch]
name=Zend Server - noarch
baseurl=http://repos.zend.com/zend-server/rpm/noarch
enabled=1
gpgcheck=1
gpgkey=http://repos.zend.com/zend.key

Install Zend Community Edition with PHP 5.3

yum install zend-server-ce-php-5.3

Firewall

Normally I just simply open ports for: Zend Server, HTTP, HTTPS, FTP, SSH.
iptables -I INPUT 5 -p tcp --dport 10081 -j ACCEPT
iptables -I INPUT 5 -p tcp --dport 443 -j ACCEPT
iptables -I INPUT 5 -p tcp --dport 80 -j ACCEPT
iptables -I INPUT 5 -p tcp --dport 21 -j ACCEPT
iptables -I INPUT 5 -p tcp --dport 22 -j ACCEPT
iptables -L --line-numbers
/etc/init.d/iptables save
/etc/init.d/iptables restart

Testing

Browsing http://[server url or IP address]:10081 to set up Zend Server.

Others

To restart Zend Server:

/etc/init.d/zend-server restart

To add Zend Server to server start up:

chkconfig zend-server on

Install MYSQL server and PhpMyAdmin

Install MYSQL client and server:

yum install mysql mysql-server

Start MYSQL server & Initialisation


/etc/init.d/mysqld start

mysql_secure_installation

Add MYSQL to server start up service

chkconfig mysqld on

Install PhpMyAdmin


rpm -Uvh http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm

yum install phpmyadmin

Add your IP to allow list in phpMyAdmin apache conf file:

vim /etc/httpd/conf.d/phpMyAdmin.conf

/etc/init.d/zend-server restart

Install Magento

Download Magento Sample data

Use phpMyAdmin to create Magento database and import the sample database dump.

wget http://www.magentocommerce.com/downloads/assets/1.6.1.0/magento-sample-data-1.6.1.0.tar.gz</pre>
tar -zxvf magento-sample-data-1.6.1.0.tar.gz

Move sample media files to web folder (I use /var/www/html as document root folder here):


mv magento-sample-data-1.6.1.0/media /var/www/html/

Download Magento


wget http://www.magentocommerce.com/downloads/assets/1.6.1.0/magento-1.6.1.0.tar.gz

tar -zxvf magento-1.6.1.0.tar.gz

cp -r magento/* /var/www/html/

cp magento/.htaccess /var/www/html/

change folder permission


chmod -R o+w media var

chmod o+w app/etc

Set up FTP

I will need a FTP account to deploy my Magento skins to server. Proftpd is clearly a better choice.


yum install proftpd

service proftpd start

chkconfig proftpd on

useradd magento
 passwd magento

Need to change user's default login place:

vim /etc/passwd 

Give user written permission to Magento folder:

chown magento.magento /var/www/html/ -Rf 

Enable Web URL Rewrite to support SEO

vim /etc/httpd/conf/httpd.conf

Change AllowOverride All for your document folder, restart Zend Server.

Now proceed with Magento GUI installation by browsing: http://[server url]

Reference:

http://files.zend.com/help/Zend-Server-Community-Edition/zend-server-community-edition.htm#rpm_installation.htm

http://www.magentocommerce.com/wiki/groups/227/installing_magento_via_shell_ssh

Updates

I found folder permission is not enough for Magento Connect to use, on the other side, I need to make sure FTP user has permission to upload source files. Here is the steps I found works well without setting folder permission to 777.

#add apache to FTP user group: magento
#your magento folder should have magento.magento ownership
#while you are in magento folder, do following:
sudo usermod -G magento apache
sudo find . -type f -exec chmod 644 {} \;
sudo find . -type d -exec chmod 775 {} \;
sudo chmod o+w var var/.htaccess app/etc
sudo chmod 550 lib/PEAR
sudo chmod -R o+w media
sudo /etc/init.d/httpd restart

My Favorite Centos Installation

I have installed many Linux systems, from Fedora, and now fall in love with Centos.

The best thing is you can install a lot of popular software just via YUM.

Here I just want to share that RPMForge is almost a must Repos you should add into your YUM.

It gives you a lot of useful packages that doesn’t come with default, such as phpmyadmin.

For how to install RPMForge, just follow this page.

On the other hand, if you would need latest version on PHP53, MYSQL etc, you should consider IUS Community.

It is proudly supported by RackSpace and comes with a lot of latest version packages.

For how to get started with IUS, just follow their Client User Guide.

Another tips to install latest version of PHP from IUS is to install php53u instead of php.

Server Time Issue

In Australia, we have day light saving in summer, which confuses a lot of my servers’ timing. To make it easier, just set up the NTP service and configure the right timezone.

Timezone:

  1. make a backup: mv /etc/localtime /etc/localtime.bak
  2. cp /usr/share/zoneinfo/Australia/Sydney /etc/localtime

Set up NTP:

  1. yum install ntp
  2. chkconfig ntpd on
  3. ntpdate 0.au.pool.ntp.org
  4. /etc/init.d/ntpd start

Please note that if you are doing the time setting on virtual system, like XenServer, you should change time on host because normally the guest system’s default time setting is locked with host.

http://www.topologi.com/

Install PHPMyAdmin on Centos 5.5 64bit

Just set up a server (VPS) for customer and realized that PhpMyAdmin is not available in Yum any more?

Apparently we need to follow steps below to be able to install PhpMyAdmin via Yum.

rpm --import http://dag.wieers.com/rpm/packages/RPM-GPG-KEY.dag.txt
cd /tmp
wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm
rpm -ivh rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm

The current version of rpmforge-release can be found here: http://packages.sw.be/rpmforge-release

yum install phpmyadmin

Zend Magento Installation Guide

For a successful e-commerce website, the speed is very important. That’s why we need to consider Zend Server + Magento.

Here is a quick installation guide to help you get started.

Environment: VPS from Rackspace (Centos 5.4)

  1. download zend+magento from Magento Download Page
  2. modify zend.repos.rpm in [ISV] section:  (This could be a bug in the installation script)
    before:
    http://repos.zend.com/isv/rpm/pe/noarch/
    after:
    http://repos.zend.com/isv/rpm/noarch/
  3. run ./install_zs.sh   (This will install everything for you. Apache2, Zend, MYSQL, Magento ….).
  4. Open Server firewall to allow access from port: 80 and 10081 (Zend Server Console)
    iptables -I RH-Firewall-1-INPUT 9 -p tcp –dport 80 -j ACCEPT
    iptables -I RH-Firewall-1-INPUT 9 -p tcp –dport 10081 -j ACCEPT
  5. visit http://<yourserver>:10081 to change zend server admin password
  6. Change MYSQL password:
    mysqladmin -u root password <new password>
    mysqladmin -u root -h <local hostname> <new password>
  7. Create a MYSQL user just for Magento:
    grant all privileges on magento.* to magentodbuser@localhost identified by ‘magentodbpassword*’
    vim /usr/share/magento/app/etc/local.xml
    to put new dbuser and dbpassword in Magento’s configure file
  8. I don’t know what’s default password to login to Magento Admin Panel, so I changed it directly in MYSQL:
    go to magento database to change password for admin user:
    update admin_user set password = MD5(‘temppassword’) where user_id = 1
  9. After login to Magento Admin Panel, you will get a warning message to change {baseurl}, and here we go:
    in admin panel, system configuration, web, UnSecure and Secure
    change {{base_url}} in Base URL (only) to http://yourdomainname
  10. Done.

Let me know if it helps. Thanks.

Setup Mercurial on Debian 5 with Apache

Mercurial Version Control

Mercurial has become more and more popular to developers. It has lots of advantages compared with Subversion. For more information, please read this excellent tutorial: http://hginit.com

I was asked to set up Mercurial for my clients the other day, and there isn’t much resource out there to follow. It took me one whole day to get this done. I think it is worth to share my experience here, and hopefully it will help others some day. Please let me know you have been here.

OS Selection

After researching, I found Debian has the best support for Mercurial in all Linux Platforms. I have been too familiar with Centos, Fedora and I always wanna play a little bit with Debian. Some one told me long long ago that Debian is the most stable Linux System… So I figured, well, let us go with Debian.

Install Mercurial

Debian has Mercurial package for install, but the version is quit old (1.0 or something). No worries, Debian has this great thing called: Backports where you can try lots of newer version software. Follow the Backports Instruction you can easily set up your Debian system getting packages from Backports. Now, let us install Mercurial:

apt-get -t lenny-backports install "mercurial"

You are done. Test it by running:

hg version

You should see:

Mercurial Distributed SCM (version 1.3.1)

Copyright (C) 2005-2009 Matt Mackall <mpm@selenic.com> and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

You can also try with creating a new repository:

hg init test

It will create a repository called “test” in your current folder.

Mercurial Repositories Publishing

Now we have Mercurial Central Server running, but how do you share to others. This is where Publishing comes from.
You may want to read this article to know more about Mercurial Publishing. There are many ways to publish your Mercurial Repository. I decided to go with the most popular one – Apache Web Server.

Install Apache mod_wsgi

If goes with Apache, it is recommended to use Apache mod_wsgi to talk to Mercurial. Here is the how-to instruction.

mod_wsgi doesn’t come with Apache by default. Now we need Backports’ help again.

Install Apache 2 if you haven’t done so:

apt-get install apache2

Install mod_wsgi

apt-get install libapache2-mod-wsgi

Restart Apache 2, you will find you have mod_wsgi already loaded.

Download hgwebdir.wsgi

hgwebdir.wsgi is needed for Apache to talk to Mercurial. It is in Mercurial Source package. We need to download it separately. Make sure you download the same version one. Here I just run:

wget http://backports.org/debian/pool/main/m/mercurial/mercurial_1.3.1.orig.tar.gz
tar -zxvf mercurial_1.3.1.orig.tar.gz

The hgwebdir.wsgi should be sitting in the contrib folder.

Apache Configuration

First of all, make two folders:

/var/www/vhosts/hg.yourdomain.com/htdocs
/var/www/vhosts/hg.yourdomain.com/cgi-bin

Now put hgwebdir.wsgi into /var/www/vhosts/hg.yourdomain.com/cgi-bin

New a file called hgweb.config in /var/www/vhosts/hg.yourdomain.com/cgi-bin:

[web]
style = coal
allow_push = *
push_ssl = false
[paths]
/ = /var/www/vhosts/hg.yourdomain.com/htdocs/**

That’s just for hgwebdir.wsgi to read HG configurations. You should change the setting in this .config file.

Update: In hgwebdir.wsgi there is a line that lists the path to hgweb.config. This line *must* be an absolute path!! even when the files are in the same directory. (Thanks to hg)

Time to change Apache custom config file: /etc/apache2/httpd.conf :

<VirtualHost *:80>
 ServerName hg.yourdomain.com
 DocumentRoot /var/www/vhosts/hg.yourdomain.com/htdocs
 ErrorLog /var/log/apache2/hg.yourdomain.com-error_log
 CustomLog /var/log/apache2/hg.yourdomain.com-access_log common
 WSGIScriptAliasMatch ^(.*)$ /var/www/vhosts/hg.yourdomain.com/cgi-bin/hgwebdir.wsgi$1
 # To enable "daemon" mode, uncomment following lines. (Read mod_wsgi docs for more info)
 # WSGIDaemonProcess hg.yourdomain.com user=USER group=GROUP threads=15 maximum-requests=1000
 # some more interesting options (tested on mod_wsgi 2.0):
 # processes=2 umask=0007 display-name=wsgi-hg.yourdomain.com inactivity-timeout=300
 # WSGIProcessGroup hg.yourdomain.com
 <Directory /var/www/vhosts/hg.yourdomain.com/htdocs>
 Options FollowSymlinks
 DirectoryIndex index.html
 AllowOverride None
 Order allow,deny
 Allow from all
 </Directory>
 <Directory /var/www/vhosts/hg.yourdomain.com/cgi-bin>
 Options ExecCGI FollowSymlinks
 AddHandler wsgi-script .wsgi
 AllowOverride None
 Order allow,deny
 Allow from all
 </Directory>
</VirtualHost>

Don’t panic, you are almost there.

Create Mercurial Repository

Now we can officially create a repository. You should always create in

/var/www/vhosts/hg.yourdomain.com/htdocs

for Apache2 to locate it. You will also need to change your repository folder permission to allow Apache 2 to write it.

cd /var/www/vhosts/hg.yourdomain.com/htdocs
hg init firstrepos
chown www-data.root firstrepos/ -R

Now, restart Apache

/etc/init.d/apache2 restart

Try browse:

http://hg.yourdomain.com

You should see something like:

Eclipse Mercurial

You already get your job done. This part is optional. I am Eclipse fun ~

Follow instruction on this website to set up your Eclipse with Mercurial.

Please note: Sometimes Eclipse Mercurial complains can’t find hp.exe, well, you need to manually guide it.

Trouble shooting:

1, Firewall – Did you open at least Port 80

2, Add below to your Apache 2 configuration file to avoid Apache warnings

ServerName hg.yourdomain.com

3,The Apache Log file path could be different on your system.

/var/log/apache2/hg.yourdomain.com-error_log

Feedback

English is not my first language, please forgive any errors above. Hopefully you will find it useful.

Pocket size Linux Server – SheevaPlug

I have been looking for this very small size server for one of my web applications. I am going to book one on Ebay and haven’t got it yet. Apparently we can install Apache, PHP, mysql onto this box, and run it as a web server. Since it is very small, we can embedded it into other devices and make it talk to the device through Ethernet. User just need to use browser to access the device data, because SheevaPlug collects the data from devices and presents it on the web pages.

Some useful links:

SheevaPlug Website

Marvell Company

Plugging Ahead: DIY Basic Apache/PHP Web Server by Rob Reilly

Replace String Recursively in Linux

Recently had an accident on one of our Linux Servers. It was hacked and all HTML and PHP files are appended by an iFrame HTML code. That iFrame will lead user’s browser to visit a website to download a virus.

Not sure what’s the cause, but we have changed the admin password.

Now problem is how do I remove that iFrame quickly? The answer is rpl.

Here is the command to list all the files in current folder that contain a particular string:

grep -r "rbomce" ./

Here is the command to replace the old string to new string: (add ‘-s’ for testing)

rpl -dR -x"html" -x"htm" -v 'oldstring' 'newstring' ./

In our case, oldstring will be that piece of iFrame code and newstring will be empty.

More information for rpl can be found in here.

Well, it turns out there is another way to do this (clean up the infected files by some malware):

Save below content into xrpl.sh: (oldText is where you put the string to be replaced)

#!/bin/sh
oldText='<iframe src="xxxxxx" width=0 height=0 style="h
idden" frameborder=0 marginheight=0 marginwidth=0 scrolling=no></iframe>'
grep -wrl "$oldText" $1 |
while read filename
do
(
echo $filename
sed "s|$oldText||g;" "$filename"> "$filename.xx"
mv "$filename.xx" "$filename"
)
done

Then make this script executable. (chmod +x xrpl.sh).

To run the script on a folder: ./xrpl.sh <folder name>/