hadoop 在redhat linux5 上部署成功(三机)

keren

断断续续，折腾了一个礼拜，终于在出差的时候把这个hadoop部署成功了。
我的环境是 VMware + redhat linux5，一台namenode，一台dataqnode，配置的过程中也遇到不少麻烦，一会儿再一一列举。
后来又在slaves里面加了个datanode节点，也成功了。看来这段时间的不断尝试没有白费，接下来就要开始在window+cygwin+eclipse里面写分布式搜索引擎了。
现在将成功的运行结果列出来。
[root@rh0 bin]# hadoop fs -lsr /tmpdrwxr-xr-x   - root supergroup          0 2010-11-07 09:41 /tmp/hadoop-rootdrwxr-xr-x   - root supergroup          0 2010-11-07 09:41 /tmp/hadoop-root/tmpdrwxr-xr-x   - root supergroup          0 2010-11-07 09:41 /tmp/hadoop-root/tmp/mapreddrwx-wx-wx   - root supergroup          0 2010-11-07 09:42 /tmp/hadoop-root/tmp/mapred/system[root@rh0 bin]# hadoop dfsadmin -reportConfigured Capacity: 13391486976 (12.47 GB)Present Capacity: 8943796224 (8.33 GB)DFS Remaining: 8943742976 (8.33 GB)DFS Used: 53248 (52 KB)DFS Used%: 0%Under replicated blocks: 0Blocks with corrupt replicas: 0Missing blocks: 0-------------------------------------------------Datanodes available: 1 (1 total, 0 dead)Name: 192.168.126.101:50010Decommission Status : NormalConfigured Capacity: 13391486976 (12.47 GB)DFS Used: 53248 (52 KB)Non DFS Used: 4447690752 (4.14 GB)DFS Remaining: 8943742976(8.33 GB)DFS Used%: 0%DFS Remaining%: 66.79%Last contact: Sun Nov 07 09:42:18 CST 2010[root@rh0 bin]# ./hadoop fs -put /usr/local/cjd/b.txt /tmp/hadoop-root/tmp/cjd/b.txt[root@rh0 bin]# ./hadoop jar hadoop-0.20.2-examples.jar wordcount /tmp/hadoop-root/tmp/cjd /tmp/hadoop-root/tmp/output-dir10/11/07 09:43:03 INFO input.FileInputFormat: Total input paths to process : 110/11/07 09:43:03 INFO mapred.JobClient: Running job: job_201011070941_000110/11/07 09:43:04 INFO mapred.JobClient:  map 0% reduce 0%10/11/07 09:43:15 INFO mapred.JobClient:  map 100% reduce 0%10/11/07 09:43:36 INFO mapred.JobClient:  map 100% reduce 100%10/11/07 09:43:39 INFO mapred.JobClient: Job complete: job_201011070941_000110/11/07 09:43:39 INFO mapred.JobClient: Counters: 1710/11/07 09:43:39 INFO mapred.JobClient:   Job Counters 10/11/07 09:43:39 INFO mapred.JobClient:     Launched reduce tasks=110/11/07 09:43:39 INFO mapred.JobClient:     Launched map tasks=110/11/07 09:43:39 INFO mapred.JobClient:     Data-local map tasks=110/11/07 09:43:39 INFO mapred.JobClient:   FileSystemCounters10/11/07 09:43:39 INFO mapred.JobClient:     FILE_BYTES_READ=183610/11/07 09:43:39 INFO mapred.JobClient:     HDFS_BYTES_READ=136610/11/07 09:43:39 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=370410/11/07 09:43:39 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=130610/11/07 09:43:39 INFO mapred.JobClient:   Map-Reduce Framework10/11/07 09:43:39 INFO mapred.JobClient:     Reduce input groups=13110/11/07 09:43:39 INFO mapred.JobClient:     Combine output records=13110/11/07 09:43:39 INFO mapred.JobClient:     Map input records=3110/11/07 09:43:39 INFO mapred.JobClient:     Reduce shuffle bytes=183610/11/07 09:43:39 INFO mapred.JobClient:     Reduce output records=13110/11/07 09:43:39 INFO mapred.JobClient:     Spilled Records=26210/11/07 09:43:39 INFO mapred.JobClient:     Map output bytes=205510/11/07 09:43:39 INFO mapred.JobClient:     Combine input records=17910/11/07 09:43:39 INFO mapred.JobClient:     Map output records=17910/11/07 09:43:39 INFO mapred.JobClient:     Reduce input records=131[root@rh0 bin]# hadoop dfsadmin -reportConfigured Capacity: 13391486976 (12.47 GB)Present Capacity: 8943679754 (8.33 GB)DFS Remaining: 8943587328 (8.33 GB)DFS Used: 92426 (90.26 KB)DFS Used%: 0%Under replicated blocks: 0Blocks with corrupt replicas: 0Missing blocks: 0-------------------------------------------------Datanodes available: 1 (1 total, 0 dead)Name: 192.168.126.101:50010Decommission Status : NormalConfigured Capacity: 13391486976 (12.47 GB)DFS Used: 92426 (90.26 KB)Non DFS Used: 4447807222 (4.14 GB)DFS Remaining: 8943587328(8.33 GB)DFS Used%: 0%DFS Remaining%: 66.79%Last contact: Sun Nov 07 09:46:22 CST 2010[root@rh0 bin]# hadoop fs -lsr /tmpdrwxr-xr-x   - root supergroup          0 2010-11-07 09:41 /tmp/hadoop-rootdrwxr-xr-x   - root supergroup          0 2010-11-07 09:43 /tmp/hadoop-root/tmpdrwxr-xr-x   - root supergroup          0 2010-11-07 09:42 /tmp/hadoop-root/tmp/cjd-rw-r--r--   1 root supergroup       1366 2010-11-07 09:42 /tmp/hadoop-root/tmp/cjd/b.txtdrwxr-xr-x   - root supergroup          0 2010-11-07 09:41 /tmp/hadoop-root/tmp/mapreddrwx-wx-wx   - root supergroup          0 2010-11-07 09:43 /tmp/hadoop-root/tmp/mapred/system-rw-------   1 root supergroup          4 2010-11-07 09:42 /tmp/hadoop-root/tmp/mapred/system/jobtracker.infodrwxr-xr-x   - root supergroup          0 2010-11-07 09:43 /tmp/hadoop-root/tmp/output-dirdrwxr-xr-x   - root supergroup          0 2010-11-07 09:43 /tmp/hadoop-root/tmp/output-dir/_logsdrwxr-xr-x   - root supergroup          0 2010-11-07 09:43 /tmp/hadoop-root/tmp/output-dir/_logs/history-rw-r--r--   1 root supergroup      29085 2010-11-07 09:43 /tmp/hadoop-root/tmp/output-dir/_logs/history/rh0_1289094087484_job_201011070941_0001_conf.xml-rw-r--r--   1 root supergroup       7070 2010-11-07 09:43 /tmp/hadoop-root/tmp/output-dir/_logs/history/rh0_1289094087484_job_201011070941_0001_root_word+count-rw-r--r--   1 root supergroup       1306 2010-11-07 09:43 /tmp/hadoop-root/tmp/output-dir/part-r-00000[root@rh0 bin]# hadoop fs -get /tmp/hadoop-root/tmp/output-dir /usr/local/cjd/b[root@rh0 bin]#

【加入一个节点后查看节点】已经有2个可用的了。
Datanodes available: 2 (2 total, 0 dead)

[root@rh0 bin]# hadoop dfsadmin -reportConfigured Capacity: 26782973952 (24.94 GB)Present Capacity: 18191843343 (16.94 GB)DFS Remaining: 18191613952 (16.94 GB)DFS Used: 229391 (224.01 KB)DFS Used%: 0%Under replicated blocks: 0Blocks with corrupt replicas: 0Missing blocks: 0-------------------------------------------------Datanodes available: 2 (2 total, 0 dead)Name: 192.168.126.102:50010Decommission Status : NormalConfigured Capacity: 13391486976 (12.47 GB)DFS Used: 53263 (52.01 KB)Non DFS Used: 4143476721 (3.86 GB)DFS Remaining: 9247956992(8.61 GB)DFS Used%: 0%DFS Remaining%: 69.06%Last contact: Sun Nov 07 10:06:55 CST 2010Name: 192.168.126.101:50010Decommission Status : NormalConfigured Capacity: 13391486976 (12.47 GB)DFS Used: 176128 (172 KB)Non DFS Used: 4447653888 (4.14 GB)DFS Remaining: 8943656960(8.33 GB)DFS Used%: 0%DFS Remaining%: 66.79%Last contact: Sun Nov 07 10:06:55 CST 2010

【控制台截图】



【遇到的问题】

0, 要把export HADOOP_HOME=/usr/local/cjd/hadoop/hadoop-0.20.2 加入 /etc/profile,否则datanode不知道这个？
1,"Name node is in safe mode",执行以下命令.
  bin/hadoop dfsadmin -safemode leave
  bin/hadoop dfsadmin -safemode off
2,如果修改了配置文件，然后datanode起不来，发现报如下错误，
Incompatible namespaceIDs in /tmp/hadoop-root/tmp/dfs/data: namenode namespaceID = 1952086391; datanode namespaceID = 1896626371
那么到 datanode的 /tmp/hadoop-root/tmp/dfs/data/current/文件系统下，把VERSION里面的namespaceID=xxxx 改成 前面这个：1952086391. 重启hadoop即可。
3,配置文件xml里面的机器名要用ip地址。要用ip保险一些。包括 master and slaves文件里面也写ip。
4,要在每个datanode上，telnet namenode 9000，如果能telnet通，说明没有问题。
5,wordcount启动问题：
  192.168.126.100: Address 192.168.126.100 maps to rh0, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!  
该问题的处理如下：请确保 /etc/xxx/xx/host是如下：
# that require network functionality will fail.#127.0.0.1        rh0 localhost [配置成这个会报错]127.0.0.1        localhost::1        localhost6.localdomain6 localhost6192.168.126.100 rh0192.168.126.101 rh1192.168.126.102 rh2