首页 新闻 会员 周边

Hadoop streaming使用libhdfs想hdfs写文件

0
悬赏园豆:10 [待解决问题]

我想使用libhdfs想hdfs中写入一个文件,代码如下:`

void create()
{
    hdfsFS fs = hdfsConnect("default",0 );
    const char* writePath = "/user/wang/helloworld";
    hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_RDWR||O_CREAT, 0, 0, 0);
    if(!writeFile)
    {
      fprintf(stderr, "Failed to open %s for writing!\n", writePath);
      exit(-1);
    }
    char* buffer = "Hello, World!";
    tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, strlen(buffer)+1);
    if (hdfsFlush(fs, writeFile))
    {
       fprintf(stderr, "Failed to 'flush' %s\n", writePath);
      exit(-1);
    }
hdfsCloseFile(fs, writeFile);
}

int main(int argc, char **argv)
{
    create();
}

在本地可以运行,并能创建文件,但是到Hadoop集群上运行却出错,不断地reduce,下面是程序的输出:

14/05/28 13:35:33 INFO streaming.StreamJob:  map 0%  reduce 0%
14/05/28 13:35:37 INFO streaming.StreamJob:  map 54%  reduce 0%
14/05/28 13:35:38 INFO streaming.StreamJob:  map 99%  reduce 0%
14/05/28 13:35:39 INFO streaming.StreamJob:  map 100%  reduce 0%
14/05/28 13:35:44 INFO streaming.StreamJob:  map 100%  reduce 1%
14/05/28 13:35:45 INFO streaming.StreamJob:  map 100%  reduce 9%
14/05/28 13:35:46 INFO streaming.StreamJob:  map 100%  reduce 16%
14/05/28 13:35:47 INFO streaming.StreamJob:  map 100%  reduce 23%
14/05/28 13:35:49 INFO streaming.StreamJob:  map 100%  reduce 20%
14/05/28 13:35:50 INFO streaming.StreamJob:  map 100%  reduce 13%
14/05/28 13:35:51 INFO streaming.StreamJob:  map 100%  reduce 9%
14/05/28 13:35:52 INFO streaming.StreamJob:  map 100%  reduce 0%
14/05/28 13:35:55 INFO streaming.StreamJob:  map 100%  reduce 3%
14/05/28 13:35:56 INFO streaming.StreamJob:  map 100%  reduce 10%
14/05/28 13:35:57 INFO streaming.StreamJob:  map 100%  reduce 16%
14/05/28 13:35:59 INFO streaming.StreamJob:  map 100%  reduce 20%
14/05/28 13:36:00 INFO streaming.StreamJob:  map 100%  reduce 21%
14/05/28 13:36:01 INFO streaming.StreamJob:  map 100%  reduce 20%
14/05/28 13:36:02 INFO streaming.StreamJob:  map 100%  reduce 13%
14/05/28 13:36:03 INFO streaming.StreamJob:  map 100%  reduce 7%
14/05/28 13:36:04 INFO streaming.StreamJob:  map 100%  reduce 0%
14/05/28 13:36:07 INFO streaming.StreamJob:  map 100%  reduce 4%
14/05/28 13:36:08 INFO streaming.StreamJob:  map 100%  reduce 10%
14/05/28 13:36:09 INFO streaming.StreamJob:  map 100%  reduce 17%
14/05/28 13:36:10 INFO streaming.StreamJob:  map 100%  reduce 23%
14/05/28 13:36:11 INFO streaming.StreamJob:  map 100%  reduce 22%
14/05/28 13:36:12 INFO streaming.StreamJob:  map 100%  reduce 18%
14/05/28 13:36:13 INFO streaming.StreamJob:  map 100%  reduce 11%
14/05/28 13:36:14 INFO streaming.StreamJob:  map 100%  reduce 5%
14/05/28 13:36:15 INFO streaming.StreamJob:  map 100%  reduce 0%
14/05/28 13:36:18 INFO streaming.StreamJob:  map 100%  reduce 5%
14/05/28 13:36:19 INFO streaming.StreamJob:  map 100%  reduce 11%
14/05/28 13:36:20 INFO streaming.StreamJob:  map 100%  reduce 18%
14/05/28 13:36:21 INFO streaming.StreamJob:  map 100%  reduce 21%
14/05/28 13:36:22 INFO streaming.StreamJob:  map 100%  reduce 100%
14/05/28 13:36:22 INFO streaming.StreamJob: To kill this job, run:
14/05/28 13:36:22 INFO streaming.StreamJob: /home/hadoop/cloudera/hadoop-0.20.2-cdh3u6/bin/hadoop job  -Dmapred.job.tracker=hadoop11.centos64:9001 -kill job_201405200934_0570
14/05/28 13:36:22 INFO streaming.StreamJob: Tracking URL:      http://hadoop11.centos64:50030/jobdetails.jsp?jobid=job_201405200934_0570
14/05/28 13:36:22 ERROR streaming.StreamJob: Job not successful. Error: NA
14/05/28 13:36:22 INFO streaming.StreamJob: killJob...
Streaming Command Failed!

运行脚本如下:

hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming-0.20.2-cdh3u6.jar  -file mapper -file reducer -mapper ./mapper -reducer ./reducer -input $1 -output $2

我是刚接触Hadoop,不是很懂,所以有请给位多指教,急用!!!

没出没的主页 没出没 | 初学一级 | 园豆:8
提问于:2014-05-28 15:03
< >
分享
清除回答草稿
   您需要登录以后才能回答,未注册用户请先注册