Linux “: command not found” Error – Windows Line Break

Today, I am working on a project, that uses Puppet template to create a stackdriver config file: /etc/sysconfig/stackdriver

For some reason, this Puppet template (.erb) is ‘dos’ file format (from my colleague), which is Windows format. The line break is CR + LF. Usually this is OK, because the config file mostly just for Linux service to read. However, in stackdriver service startup script, there is a line ‘ . /etc/sysconfig/stackdriver ‘. Stackdrive service uses this config file to initial certain variables. In this case, if you have Windows line breaks in the config file, you will see this error:

: command not found
: command not found

I then use sublime text to recreate this file in unix file format, problem goes away. Hope this helps someone.

These sublime text personal preference setting are recommended, unless you are a Windows platform developer:

{
"default_line_ending": "unix",
"translate_tabs_to_spaces": true
}

Update: It turns out it is because I copied the files from my Windows PC over to Linux server. What I should do is to git clone the files on the Linux server. This way Git will help deal with the line ending issue.

Debug SSH Connection (Password less)

When you are trying set up SSH access to an AWS instance with your private key, sometimes you may run into errors:

Received disconnect from xxx.xxx.xxx.xxx: 2: Too many authentication failures for ubuntu

The error message you get from that is not very helpful generally.

Things you can try is to add ‘-v’ to the SSH command, it will give you more debug information.

However, those debug information is still not enough and misleading.

To get an absolute real problem, you should get into the remote box and read the auth.log file – this will give you exact errors you are having, which makes this kinda of problems much easy to fix.

tail -f /var/log/auth.log

Headless Ubuntu Package Installation

With DevOps concept in mind, I don’t normally set up a server manually anymore. I either use Puppet (Chef, Ansible) or write a quick Bash script to do the server provisioning. What happens is sometimes a package I need to install may pop up a blue GUI asking for user inputs. This is annoying as it could break my server provisioning automation. No worries – I can use debconf-set-selections(Ubuntu) to preconfigure the user inputs.

Let us use an example, today I need to add newrelic-php5 to a server provisioning script written in Bash.

First let me see if I have any preconfigured values in place for newrelic-php5:

# debconf-get-selections |grep newrelic
0

Good, let me go ahead and install it.

# apt-get install newrelic-php5 -y

newrelic_php5_headless_installation

As you can see, it asks me for newrelic license and followed by application name. After filling the details and finishing the installation, I run this command again to see if there is any preconfigured values saved in debconf database:

# debconf-get-selections |grep newrelic
newrelic-php5 newrelic-php5/5.1-deprecation note
newrelic-php5 newrelic-php5/application-name string PHP Application
newrelic-php5 newrelic-php5/license-key string xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

This time I got something. To install this package headless, since I was asked to two inputs during the installation, one for license and one for application name, all I need to do is to preseed these two values with debconf-set-selections.

Let me start it over.

First, remove the package and clean up the debconf:
# apt-get remove newrelic-php5 -y
# echo PURGE | debconf-communicate newrelic-php5

OK, let me preseed the values:
# debconf-set-selections <<< "newrelic-php5 newrelic-php5/application-name string myappname" # debconf-set-selections <<< "newrelic-php5 newrelic-php5/license-key string xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

Verify they are being set:
# debconf-get-selections |grep newrelic
newrelic-php5 newrelic-php5/application-name string myappname
newrelic-php5 newrelic-php5/license-key string xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Try install it again:

# apt-get install newrelic-php5 -y

Boom. It works. Happy scripting.

If your website consumes 10TB bandwidth per month, you should use CloudFlare

In our example, we are running couple of high traffic websites on AWS EC2. Monthly data transfer out is more than 30TB. Just the bandwidth charges from AWS is almost $4000.

With CloudFlare in the front and caching all our assets (Images, Css and JS), it saves us over 20TB bandwidth per month. That’s more than $2500 savings.

Not to mention that you can get tons of other great features from CloudFlare, such as Web Application Firewall.

Worth mentioning that CloudFront from AWS charges same bandwidth rate as AWS EC2, so THUMB down to CloudFront & THUMB up to CloudFlare.

CloudFlare

Bash – set variable with or without export are different

I am working on an AWS ec2 instance provisioning script today. I am using Packer to provision the instance and build an AMI. The instance is running Ubuntu. There is a very annoying error when I am using ‘apt-get -y upgrade’ or ‘apt-get -y install':

debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin:

Google it and not hard to find the fix is to add “DEBIAN_FRONTEND=noninteractive” in your script
(credit to link to stackoverflow)

However, it still doesn’t work for me until I realize I need to put in ‘export':

export DEBIAN_FRONTEND=noninteractive

Apparently ‘export’ will make the variable is available to any process you run from your current shell process (not parent process though). Without ‘export’, the variable scope is restricted to the shell, and is not available to any other process.
(credit to link to stackoverflow)

When you need more disk space for AWS WorkDoc on AWS WorkSpace

As of today, AWS WorkSpace (Standard) only comes with 50G user data (D drive). You can install WorkDoc client on your WorkSpace to have your data synced. However, if your WorkDoc data exceed 50G, this will fill up your WorkSpace D drive very quickly.

Here a work around:

  • Start a EC2 Windows instance with a big enough EBS, say 300G. This instance must be launched in the same VPC as your WorkSpace.
  • Join this instance to your AWS Directory, same Directory as your WorkSpace
  • Install WorkDoc client on this EC2 instance and share the WorkDoc folder to Directory/Users
  • Added this instance to the existing WorkSpace Controller security group or make sure it is accessible to WorkSpaces via Port 445
  • In each WorkSpace, just map the shared drive on this EC2 instance. \\[ec2 instance computer name]\workdoc

There you have it, good speed, plenty of spaces for your WorkDoc data.

For how to join EC2 instance to your AWS Directory, please refer to this Doc (I am in AWS Sydney region, and we can only do this manually): http://docs.aws.amazon.com/directoryservice/latest/adminguide/join_windows_instance.html

You will also be able to add Directory Management tools on this EC2 instance for management purpose.
http://docs.aws.amazon.com/directoryservice/latest/adminguide/install_ad_tools.html

Hope this helps. If you have other ideas and suggestions, please feel free to let me know.

Set up an Amazon Ubuntu server to run Zend Framework PHP Web Application

First of all, I am new to Ubuntu.

I am a Centos fan for last couple of years, but have heard of Ubuntu is a great system too, especially user friendly.

Today, one client asked me to rebuild a Amazon server for his Zend Framework PHP Web Application dut to its current stability issue. (Amazon Linux seems to have issue with MySQL which causes MySQL memory issue randomly). I figured it is a good time for me to try Ubuntu.

First, start up a new Amazon micro instance with Ubuntu LTS (LTS seems to be more stable and long term support).

Install Zend Server 6.0 Community Edition

Zend Server CE has made the installation so easy for us.

  1. SSH to server
  2. download Zend Server (DEB/RPM Installer Script)
  3. Unzip it and run the install script as root:
    ./install_zs.sh 5.3     or    ./install_zs.sh 5.4  if you need 5.4 version PHP
  4. Simply ‘y’ when prompt
  5. Done – navigate to http://<server ip> you will see “It works”
Install MySQL
  1. apt-get install mysql-server
  2. Please be noted the default MySQL sock is in here: /var/run/mysqld/mysqld.sock

Install phpMyAdmin

  1. Navigate to http://<server ip>:10081 and set up the Zend Server
  2. Deploy Application
  3. Follow this link to download phpMyAdmin package and install it via Zend Server
  4. Well, you may want to disable the auto login for security reason, change auth_type to http in this file:
    /usr/local/zend/var/apps/http/__default__/0/phpMyAdmin/3.5.6.0/config.inc.php

Set up FTP

  1. Follow this link

“File path too long” error when copying files in Windows SBS

It was painful when you need to copy large amount of data from one server to the other and came across this “file path too long” error. That means you will need to dig into each subfolder to either rename the file or zip them up. That would take all night if you have a lot of this errors. I had this problem today.

What I am doing is helping a client migrate Windows SBS 2008 to Windows SBS 2011. There are over 200G data to copy across. For the first couple of “file path too long” errors, I just mark them down until I find there are too many of them.

Quick search on Google leads me to RoboCopy . I have to say this is an excellent tool to handle exactly this case. Actually people are using this tool to do server backup too.

robocopy "\\server1\folder1" "D:\folder2" /e /z /dcopy:T /XO

A simple command line came to rescue. I can then sit back and enjoy my coffee.

Update: Just realise today that you can use robocopy to delete files as well:
create a empty folder and run this command:

robocopy e:\empty e:\to_delete /MIR /E

Set up Unison for Two Way Syncing on 2 Centos Servers

Have been played with both Async and Unison, I found Unison is more powerful, especially on 2 way syncing include deletion. Unison is easy to get start as well.

You can setup Unison by compiling the source, but it is not as easy as I thought – I ran into some libraries dependency conflicts.

Luckily, I am working on Centos servers. I can just use Yum to install it.

Enable Fedora EPEL repos for your Yum with protectbase

wget http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
rpm -Uvh epel-release-5-4.noarch.rpm
yum install yum-protectbase.noarch -y
yum repolist

Install Unison via Yum

yum install unison -y

Enable SSH Access without Password

On local host, do these:

ssh-keygen
#followed by three ENTERs
ssh-copy-id -i .ssh/id_rsa.pub root@remotehost
ssh root@remotehost

Sample Unison Command

/usr/bin/unison /web/live1 ssh://remoteserver/web/live2 \
-ignore "Path WEB-INF/config" \
-ignore "Path WEB-INF/derby" \
-ignore "Path WEB-INF/test/test.xml"

Reference

HowTo Use unison File Synchronizer
3 Steps to Perform SSH Login Without Password Using ssh-keygen & ssh-copy-id

Centos 6.2 Net Install vs Minimal Install

The other day, I was asked to set up Zabbix monitor system for a client. Quickly checked up, it runs well in Centos – my favour Linux. Let us get started.

I always prefer to use latest version of Centos, as the date of this article, Centos 6.2 just came out.

To avoid big download bandwidth to the client’s office, I was looking two options: 1, Centos 6.2 Minimal  2, Centos 6.2 Net Install

You can find both versions for download in here: (ISO)

http://isoredirect.centos.org/centos/6/isos/i386/

Centos 6.2 Minimal Installation

The ISO only 200+MB. With this ISO, you can install a very very basic running Centos system, not even the base tools. You are mostly like need to ‘yum groupinstall base’ after system boots up. I would go this option, but I don’t want to waste too much time wondering which packages are missing and reinstalling them.

PS. During the installation, you won’t be asked to choose packages to install. You can only install more packages with Yum after.

Centos 6.2 Net Installation

ISO is 100+MB. You will need network for sure. During installation, it will ask you to config the network (or just DHCP). You will also need to put in the Centos 6.2 online images: [Your nearest Centos image mirror]/centos/6.2/os/i386/

The installation took me a while and you will be asked to choose couple of installation types: Desktop, Minimal, Basic Server …. For the difference, please see this link: CentOS 6 “Default” Installation Options

I chose “Basic Server”. After installation, the whole system is about 1.5G.

Hope this would help some one with the Centos 6 installation.