一切福田,不離方寸,從心而覓,感無不通。

windows安装hadoop

在Windows下利用cygwin仿unix环境安装配置Hadoop。

子猴也是刚接触到hadoop,对其的配置第一次按照网上的一些说明配置成功了,但有些东西感到不是很清晰,所以又重新把整个过程跑了一遍并记录下来,也是想对整个过程有个清晰的脉络,不正确之处请指教。

1、  所需软件

1.1、Cygwin(截至到目前最新版本是2.685)

下载地址:http://www.cygwin.com/setup.exe

1.2、JDK 1.6.x

1.3、hadoop-0.20.1

下载地址:http://apache.freelamp.com/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz

2、  安装

2.1、Cygwin安装说明见文章:http://hi.baidu.com/xiaoduo170/blog/item/9a21e38269a89ea46e81196f.html

补充:cygwin的bash是无法复制粘贴的,很不方便,所以可采用putty,下载地址是:

http://www.linuxboy.net/linux/rc/puttycyg.zip ,将puttycyg.zip解压后的三个exe文件放到Cygwin安装目录HOME_PATH下bin目录下,然后修改HOME_PATH下的Cygwin.bat文件,建议用记事本打开,然后将bash –login –i注释掉,在前面加rem,也就是rem bash –login –i,或者:: bash –login –i,加入 start putty -cygterm – 即可。

这样一来就可以复制粘贴了,但注意的是默认的根目录是Cygwin的HOME_PATH,如果要切换到其他主目录,但如果你想要进入到其他根目录,但如果你想要进入到其他根目录,需要通过系统根目录,子猴这里的是/cygdrive,比如要进入到e盘,则为/cygdrive/e。

2.2、JDK的安装省略了

2.3、hadoop-0.20.1安装

将hadoop-0.20.1.tar.gz解压,解压后的目录如hadoop-0.20.1,假设是放在E盘:

E:\hadoop-0.20.1,修改conf/hadoop-env.sh文件,将export JAVA_HOME的值修改为你机上的jdk安装目录,比如/cygdrive/d/tools/jdk1.6.0_03,/cygdrive是Cygwin安装成功后系统的根目录

3、  安装和配置ssh

3.1、安装

在Cygwin的根目录下分别运行:

$ chmod +r /etc/group$ chmod +r /etc/passwd$ chmod +rwx /var$ ssh-host-config*** Info: Generating /etc/ssh_host_key*** Info: Generating /etc/ssh_host_rsa_key*** Info: Generating /etc/ssh_host_dsa_key*** Info: Creating default /etc/ssh_config file*** Info: Creating default /etc/sshd_config file*** Info: Privilege separation is set to yes by default since OpenSSH 3.3.*** Info: However, this requires a non-privileged account called 'sshd'.*** Info: For more info on privilege separation read /usr/share/doc/openssh/README.privsep.*** Query: Should privilege separation be used? (yes/no) yes*** Info: Note that creating a new user requires that the current account have*** Info: Administrator privileges.  Should this script attempt to create a*** Query: new local account 'sshd'? (yes/no) yes*** Info: Updating /etc/sshd_config file*** Info: Added ssh to C:\WINDOWS\system32\driversc\services*** Info: Creating default /etc/inetd.d/sshd-inetd file*** Info: Updated /etc/inetd.d/sshd-inetd*** Warning: The following functions require administrator privileges!*** Query: Do you want to install sshd as a service?*** Query: (Say "no" if it is already installed as a service) (yes/no) yes*** Query: Enter the value of CYGWIN for the daemon: [] cygwin注:此处输入的cygwin可以是任意的)*** Info: The sshd service has been installed under the LocalSystem*** Info: account (also known as SYSTEM). To start the service now, call*** Info: net start sshd' or cygrunsrv -S sshd'.  Otherwise, it*** Info: will start automatically after the next reboot.*** Info: Host configuration finished. Have fun!

在询问yes/no的地方,统一输入yes,sshd就安装好了。

3.2配置

3.2.1、启动sshd服务

net start sshd

CYGWIN sshd 服务正在启动

CYGWIN sshd 服务已经启动成功

3.2.2、$ ssh localhost

试着连接本机看看,注意,如果在没有启动sshd服务,这个连接肯定是失败的!关于此错误也可参见:
http://www.zihou.me/2010/02/19/1521/

如果没问题,会出现下面一些内容:

The authenticity of host 'localhost (127.0.0.1)' can't be established.RSA key fingerprint is 08:03:20:43:48:39:29:66:6e:c5:61:ba:77:b2:2f:55.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added 'localhost' (RSA) to the list of known hosts.zihou@localhost's password:

会提示输入你机子的登录密码,输入无误后,会出现文本图形,类似于欢迎的提示:

The Hippo says: Welcome to

3.2.3、建立ssh的通道

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsaGenerating public/private dsa key pair.Your identification has been saved in /home/zihou/.ssh/id_dsa.Your public key has been saved in /home/zihou/.ssh/id_dsa.pub.The key fingerprint is:6d:64:8e:a6:38:73:ab:c5:ce:71:cd:df:a1:ca:63:54 zihou@PC-04101515The key's randomart image is:+--[ DSA 1024]—-+|                 ||                 ||          o      ||         *  E    ||        S +.     ||     o o +.      ||    + * ..o   .  ||     B + .o. o . ||    ..+  .ooo .  |+—————--+

$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

再执行遍$ ssh localhost看看,如果没有问题,就说明sshd已经配置好了。

4、  配置hadoop

编辑conf/hadoop-site.xml

加入以下内容:

<configuration><property><name>fs.default.name</name><value>localhost:9000</value></property><property><name>mapred.job.tracker</name><value>localhost:9001</value></property><property><name>dfs.replication</name><value>1</value></property></configuration>

5、  运行hadoop

进入到E:\hadoop-0.20.1,在cygwin下的操作如:/cygdrive/e/ hadoop-0.20.1,执行:

bin/hadoop namenode –format格式化一个新的分布式文件系统,提示信息如下:
10/02/19 17:32:26 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath.
Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml
to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
(这段我还不是很清楚,我用的最新版本)

10/02/19 17:32:26 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = PC-04101515/192.168.0.14

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 0.20.1

STARTUP_MSG:   build =

http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1 -r 810220; compiled by ‘oom’ on Tue Sep  1 20:55:56 UTC 2009

************************************************************/

10/02/19 17:32:27 INFO namenode.FSNamesystem:

fsOwner=zihou,None,root,Administrators,Users

10/02/19 17:32:27 INFO namenode.FSNamesystem: supergroup=supergroup

10/02/19 17:32:27 INFO namenode.FSNamesystem: isPermissionEnabled=true

10/02/19 17:32:28 INFO common.Storage: Image file of size 102 saved in 0 seconds.

10/02/19 17:32:28 INFO common.Storage: Storage directory \tmp\hadoop-SYSTEM\dfs\name has been successfully formatted.

10/02/19 17:32:28 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at PC-04101515/192.168.0.14

************************************************************/
6、  启动hadoop守护进程

$ bin/start-all.shstarting namenode, logging to/cygdrive/e/hadoop-0.20.1/bin/../logs/hadoop-zihou-namenode-PC-04101515.outlocalhost: datanode running as process 5200. Stop it first.localhost: secondarynamenode running as process 1664. Stop it first.starting jobtracker, logging to/cygdrive/e/hadoop-0.20.1/bin/../logs/hadoop-zihou-jobtracker-PC-04101515.outlocalhost: starting tasktracker, logging to/cygdrive/e/hadoop-0.20.1/bin/../logs/hadoop-zihou-tasktracker-PC-04101515.out

(注:如果你第一次启动,提示信息或许会与上面有所不同,我为了写这篇文章,重新执行了一遍)

7、  测试

单机模式的操作方法

下面的实例将已解压的 conf 目录拷贝作为输入,查找并显示匹配给定正则表达式的条目。输出写入到指定的output目录。(注:根目录是hadoop的目录)

$ mkdir input

$ cp conf/*.xml input

$ bin/hadoop jar hadoop-*-examples.jar grep input output ‘dfs[a-z.]+’

$ cat output/*

通过执行$ bin/hadoop dfs –ls来看是否将*.xml文件拷贝到input中了,执行后结果如下:

Found 1 items

drwxr-xr-x   – zihou supergroup          0 2010-02-19 17:44 /user/zihou/input

表示已经拷贝过去了。

在伪分布式模式上运行

bin/hadoop jar hadoop-*-examples.jar grep input output ‘dfs[a-z.]+’

如果没有错误的话,会给出一堆信息,如:
10/02/19 14:56:07 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site

.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively

10/02/19 14:56:08 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=

10/02/19 14:56:09 INFO mapred.FileInputFormat: Total input paths to process : 5

10/02/19 14:56:10 INFO mapred.JobClient: Running job: job_local_0001

10/02/19 14:56:10 INFO mapred.FileInputFormat: Total input paths to process : 5

10/02/19 14:56:10 INFO mapred.MapTask: numReduceTasks: 1

10/02/19 14:56:10 INFO mapred.MapTask: io.sort.mb = 100

10/02/19 14:56:10 INFO mapred.MapTask: data buffer = 79691776/99614720

10/02/19 14:56:10 INFO mapred.MapTask: record buffer = 262144/327680

。。。。。。。。。。。。。。。
这样,hadoop就成功配置了!

说明

Hadoop中文文档地址:http://hadoop.apache.org/common/docs/r0.18.2/cn/

快速安装说明手册:http://hadoop.apache.org/common/docs/r0.18.2/cn/quickstart.html

Hadoop简介

Hadoop是一个开放源代码的分布式文件系统,属于Apache中的一个项目,所谓分布式文件系统(Distributed File System),指的是具有执行远程文件存取的能力,并以透明方式对分布在网络上的文件进行管理和存取,客户端访问的时候不需要知道文件真正存放在哪里。 Hadoop最初是包含在Nutch中的,后来,Nutch中实现的NDFS和MapReduce代码剥离出来成立了一个新的开源项目,这就是 Hadoop。

转自:http://hi.baidu.com/xiaoduo170/item/b3c63ceda8f2720f560f1d5b