我想使用libhdfs想hdfs中写入一个文件,代码如下:`
void create() { hdfsFS fs = hdfsConnect("default",0 ); const char* writePath = "/user/wang/helloworld"; hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_RDWR||O_CREAT, 0, 0, 0); if(!writeFile) { fprintf(stderr, "Failed to open %s for writing!\n", writePath); exit(-1); } char* buffer = "Hello, World!"; tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, strlen(buffer)+1); if (hdfsFlush(fs, writeFile)) { fprintf(stderr, "Failed to 'flush' %s\n", writePath); exit(-1); } hdfsCloseFile(fs, writeFile); } int main(int argc, char **argv) { create(); }
在本地可以运行,并能创建文件,但是到Hadoop集群上运行却出错,不断地reduce,下面是程序的输出:
14/05/28 13:35:33 INFO streaming.StreamJob: map 0% reduce 0% 14/05/28 13:35:37 INFO streaming.StreamJob: map 54% reduce 0% 14/05/28 13:35:38 INFO streaming.StreamJob: map 99% reduce 0% 14/05/28 13:35:39 INFO streaming.StreamJob: map 100% reduce 0% 14/05/28 13:35:44 INFO streaming.StreamJob: map 100% reduce 1% 14/05/28 13:35:45 INFO streaming.StreamJob: map 100% reduce 9% 14/05/28 13:35:46 INFO streaming.StreamJob: map 100% reduce 16% 14/05/28 13:35:47 INFO streaming.StreamJob: map 100% reduce 23% 14/05/28 13:35:49 INFO streaming.StreamJob: map 100% reduce 20% 14/05/28 13:35:50 INFO streaming.StreamJob: map 100% reduce 13% 14/05/28 13:35:51 INFO streaming.StreamJob: map 100% reduce 9% 14/05/28 13:35:52 INFO streaming.StreamJob: map 100% reduce 0% 14/05/28 13:35:55 INFO streaming.StreamJob: map 100% reduce 3% 14/05/28 13:35:56 INFO streaming.StreamJob: map 100% reduce 10% 14/05/28 13:35:57 INFO streaming.StreamJob: map 100% reduce 16% 14/05/28 13:35:59 INFO streaming.StreamJob: map 100% reduce 20% 14/05/28 13:36:00 INFO streaming.StreamJob: map 100% reduce 21% 14/05/28 13:36:01 INFO streaming.StreamJob: map 100% reduce 20% 14/05/28 13:36:02 INFO streaming.StreamJob: map 100% reduce 13% 14/05/28 13:36:03 INFO streaming.StreamJob: map 100% reduce 7% 14/05/28 13:36:04 INFO streaming.StreamJob: map 100% reduce 0% 14/05/28 13:36:07 INFO streaming.StreamJob: map 100% reduce 4% 14/05/28 13:36:08 INFO streaming.StreamJob: map 100% reduce 10% 14/05/28 13:36:09 INFO streaming.StreamJob: map 100% reduce 17% 14/05/28 13:36:10 INFO streaming.StreamJob: map 100% reduce 23% 14/05/28 13:36:11 INFO streaming.StreamJob: map 100% reduce 22% 14/05/28 13:36:12 INFO streaming.StreamJob: map 100% reduce 18% 14/05/28 13:36:13 INFO streaming.StreamJob: map 100% reduce 11% 14/05/28 13:36:14 INFO streaming.StreamJob: map 100% reduce 5% 14/05/28 13:36:15 INFO streaming.StreamJob: map 100% reduce 0% 14/05/28 13:36:18 INFO streaming.StreamJob: map 100% reduce 5% 14/05/28 13:36:19 INFO streaming.StreamJob: map 100% reduce 11% 14/05/28 13:36:20 INFO streaming.StreamJob: map 100% reduce 18% 14/05/28 13:36:21 INFO streaming.StreamJob: map 100% reduce 21% 14/05/28 13:36:22 INFO streaming.StreamJob: map 100% reduce 100% 14/05/28 13:36:22 INFO streaming.StreamJob: To kill this job, run: 14/05/28 13:36:22 INFO streaming.StreamJob: /home/hadoop/cloudera/hadoop-0.20.2-cdh3u6/bin/hadoop job -Dmapred.job.tracker=hadoop11.centos64:9001 -kill job_201405200934_0570 14/05/28 13:36:22 INFO streaming.StreamJob: Tracking URL: http://hadoop11.centos64:50030/jobdetails.jsp?jobid=job_201405200934_0570 14/05/28 13:36:22 ERROR streaming.StreamJob: Job not successful. Error: NA 14/05/28 13:36:22 INFO streaming.StreamJob: killJob... Streaming Command Failed!
运行脚本如下:
hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming-0.20.2-cdh3u6.jar -file mapper -file reducer -mapper ./mapper -reducer ./reducer -input $1 -output $2
我是刚接触Hadoop,不是很懂,所以有请给位多指教,急用!!!