用正则表达式匹配
你把采集到的html发上来,然后指出你要求采集里面的哪些内容,你希望采集的内容有什么约束,比如上面的apparel,是不是必须在class名称为lists的div下的前面有dl和dt的a标签内容。这样才能写的。
@LCM:HTML就这多啊,采集每个</a>标签前面的单词啊 必须在class名称为lists的div下的前面有dl和dt的a标签内容
@LCM: 我还要将他们一次插入数据库 一数组的方式采集出来
@miloss: 我的意思是你把html用字符串的形势放上面,否则别人给你写的话连测试都不好测试,还得照着图上写一次那么长的字符串,累不累啊。
@LCM:
<div class="lists"> | |
<dl><dt><a href="/categories/apparel/">Apparel</a></dt><dd><a href="/wholesale-apparel/">Apparel</a></dd><dd><a href="/wholesale-blazers+sport+coats/">Blazers & Sport Coats</a></dd><dd><a href="/wholesale-costume+cosplay/">Costume & Cosplay</a></dd><dd><a href="/wholesale-hoodies+sweatshirts/">Hoodies Sweatshirts </a></dd><dd><a href="/wholesale-jeans/">Jeans</a></dd><dd><a href="/wholesale-men+s+clothing/">Men's Clothing</a></dd><dd><a href="/wholesale-men+s+sweaters/">Men's Sweaters</a></dd><dd><a href="/wholesale-other+apparel/">Other Apparel</a></dd><dd><a href="/wholesale-other+men+s+clothing/">other men's clothing</a></dd><dd><a href="/wholesale-outerwear/">Outerwear</a></dd><dd><a href="/wholesale-pants/">Pants</a></dd><dd><a href="/wholesale-shirts/">Shirts</a></dd><dd><a href="/wholesale-shorts/">Shorts</a></dd><dd><a href="/wholesale-suits/">Suits</a></dd><dd><a href="/wholesale-swimmingwear/">Swimmingwear</a></dd><dd><a href="/wholesale-underwear/">Underwear</a></dd><dd><a href="/wholesale-vests/">Vests</a></dd><dd><a href="/wholesale-women+s+blazers+jackets/">Women's Blazers Jackets</a></dd><dd><a href="/wholesale-women+s+clothing/">Women's Clothing</a></dd><dd><a href="/wholesale-women+s+dresses/">Women's Dresses</a></dd><dd><a href="/wholesale-women+s+hosiery/">Women's Hosiery</a></dd><dd><a href="/wholesale-women+s+jeans/">Women's Jeans</a></dd><dd><a href="/wholesale-women+s+juniors/">Women's Juniors</a></dd><dd><a href="/wholesale-women+s+overcoat/">Women's Overcoat</a></dd><dd><a href="/wholesale-women+s+petites/">Women's Petites</a></dd><dd><a href="/wholesale-women+s+plus+sizes/">Women's Plus Sizes</a></dd><dd><a href="/wholesale-women+s+shirts+tops/">Women's Shirts Tops</a></dd><dd><a href="/wholesale-women+s+shorts/">Women's Shorts </a></dd><dd><a href="/wholesale-women+s+skirts/">Women's Skirts</a></dd><dd><a href="/wholesale-women+s+suits/">Women's Suits</a></dd><dd><a href="/wholesale-women+s+sweaters/">Women's Sweaters</a></dd><dd><a href="/wholesale-women+s+swimmingwear/">Women's Swimmingwear</a></dd><dd><a href="/wholesale-women+s+t+shirts+tank+tops/">Women's T-Shirts Tank Tops </a></dd><dd><a href="/wholesale-women+s+trousers/">Women's Trousers</a></dd><dd><a href="/wholesale-women+s+vests/">Women's Vests</a></dd></dl><dl><dt><a href="/categories/arts+and+crafts/">Arts and Crafts</a></dt><dd><a href="/wholesale-antique+imitation+crafts/">Antique Imitation Crafts</a></dd><dd><a href="/wholesale-arts+and+crafts/">Arts and Crafts</a></dd><dd><a href="/wholesale-bamboo+and+wooden+crafts/">Bamboo and Wooden Crafts</a></dd><dd><a href="/wholesale-basketry/">Basketry</a></dd><dd><a href="/wholesale-bead+art/">Bead Art</a></dd><dd><a href="/wholesale-candle+and+soap+making/">Candle and Soap Making</a></dd><dd><a href="/wholesale-ceramics+pottery/">Ceramics & Pottery </a></dd><dd><a href="/wholesale-embroidery+and+knitting/">Embroidery and knitting</a></dd><dd><a href="/wholesale-folk+and+native+crafts/">Folk and Native Crafts</a></dd><dd><a href="/wholesale-general+art+and+craft/">General Art and Craft</a></dd><dd><a href="/wholesale-jade+crafts/">Jade Crafts</a></dd><dd><a href="/wholesale-metal+and+tin+pieces/">Metal and Tin Pieces</a></dd><dd><a href="/wholesale-mosaic+and+glass+art/">Mosaic and Glass Art</a></dd><dd><a href="/wholesale-other+arts+and+crafts/">Other Arts and Crafts</a></dd><dd><a href="/wholesale-paintings+and+calligraphies/">Paintings and Calligraphies</a></dd><dd><a href="/wholesale-paper+crafts/">Paper Crafts</a></dd><dd><a href="/wholesale-plastic+and+resinic+crafts/">Plastic and Resinic Crafts</a></dd><dd><a href="/wholesale-religious+crafts/">Religious Crafts</a></dd><dd><a href="/wholesale-sculpture+and+carvings/">Sculpture and Carvings</a></dd><dd><a href="/wholesale-shell+crafts/">Shell crafts</a></dd></dl><dl><dt><a href="/categories/bags+shoes+accessories/">Bags Shoes & Accessories</a></dt><dd><a href="/wholesale-baby+shoes/">Baby Shoes</a></dd><dd><a href="/wholesale-backpacks+and+book+bags/">Backpacks and Book Bags</a></dd><dd><a href="/wholesale-bags+handbags/">Bags & Handbags</a></dd><dd><a href="/wholesale-bags+shoes+accessories/">Bags Shoes & Accessories</a></dd><dd><a href="/wholesale-brief+case/">Brief Case</a></dd><dd><a href="/wholesale-children+s+shoes/">Children's Shoes</a></dd><dd><a href="/wholesale-clutch+bags/">Clutch Bags</a></dd><dd><a href="/wholesale-cosmetic+bags/">Cosmetic Bags</a></dd><dd><a href="/wholesale-diaper+bags/">Diaper Bags</a></dd><dd><a href="/wholesale-evening+bags/">Evening Bags</a></dd><dd><a href="/wholesale-heels+pumps/">Heels Pumps</a></dd><dd><a href="/wholesale-hobos+bags/">Hobos Bags</a></dd><dd><a href="/wholesale-laptop+bag/">Laptop Bag</a></dd><dd><a href="/wholesale-loafers/">Loafers</a></dd><dd><a href="/wholesale-luggage+brief+case/">Luggage & Brief case</a></dd><dd><a href="/wholesale-luggage+bags+travel+bags/">Luggage Bags Travel Bags</a></dd><dd><a href="/wholesale-men+s+accessories/">Men's Accessories</a></dd><dd><a href="/wholesale-men+s+athletic/">Men's Athletic</a></dd><dd><a href="/wholesale-men+s+athletic+inspired/">Men's Athletic-Inspired</a></dd><dd><a href="/wholesale-men+s+bags/">Men's Bags</a></dd><dd><a href="/wholesale-men+s+belts+buckles/">Men's Belts & Buckles</a></dd><dd><a href="/wholesale-men+s+boots/">Men's Boots</a></dd><dd><a href="/wholesale-men+s+casual+comfort/">Men's Casual Comfort</a></dd><dd><a href="/wholesale-men+s+dress+shoes/">Men's Dress Shoes</a></dd><dd><a href="/wholesale-men+s+hats+caps/">Men's Hats & Caps</a></dd><dd><a href="/wholesale-men+s+sandals+clogs/">Men's Sandals & Clogs</a></dd><dd><a href="/wholesale-men+s+shoes/">Men's Shoes</a></dd><dd><a href="/wholesale-men+s+slippers/">Men's Slippers</a></dd><dd><a href="/wholesale-messenger+bag+sling/">Messenger Bag Sling</a></dd><dd><a href="/wholesale-moccasins/">Moccasins</a></dd><dd><a href="/wholesale-other+handbags/">Other Handbags</a></dd><dd><a href="/wholesale-other+men+s+accessories/">Other Men's Accessories</a></dd><dd><a href="/wholesale-other+men+s+shoes/">Other Men's Shoes</a></dd><dd><a href="/wholesale-other+women+s+accessories/">Other Women's Accessories</a></dd><dd><a href="/wholesale-other+women+s+shoes/">Other Women's shoes</a></dd><dd><a href="/wholesale-oxfords/">Oxfords</a></dd><dd><a href="/wholesale-platforms+wedges/">Platforms Wedges</a></dd><dd><a href="/wholesale-scarves/">Scarves</a></dd><dd><a href="/wholesale-shopping+bags/">Shopping Bags</a></dd><dd><a href="/wholesale-shoulders/">Shoulders</a></dd><dd><a href="/wholesale-ties/">Ties</a></dd><dd><a href="/wholesale-tote+bags/">Tote Bags</a></dd><dd><a href="/wholesale-wallets/">Wallets</a></dd><dd><a href="/wholesale-women+s+accessories/">Women's Accessories</a></dd><dd><a href="/wholesale-women+s+athletic/">Women's Athletic</a></dd><dd><a href="/wholesale-women+s+athletic+inspired/">Women's Athletic-Inspired</a></dd><dd><a href="/wholesale-women+s+belts+buckles/">Women's Belts & Buckles</a></dd><dd><a href="/wholesale-women+s+boots/">Women's Boots</a></dd><dd><a href="/wholesale-women+s+flats/">Women's Flats</a></dd><dd><a href="/wholesale-women+s+hats+caps/">Women's Hats & Caps</a></dd><dd><a href="/wholesale-women+s+sandals/">Women's Sandals</a></dd><dd><a href="/wholesale-women+s+shoes/">Women's Shoes</a></dd><dd><a href="/wholesale-women+s+slippers/">Women's Slippers</a></dd></dl><dl><dt><a href="/categories/car+vehicles+and+accessories/">Car,Vehicles and Accessories</a></dt><dd><a href="/wholesale-boat/">boat</a></dd><dd><a href="/wholesale-car+audio+and+video/">Car Audio and Video</a></dd><dd><a href="/wholesale-car+badge/">Car Badge</a></dd><dd><a href="/wholesale-car+gps/">Car GPS</a></dd><dd><a href="/wholesale-car+maintenance/">Car Maintenance</a></dd><dd><a href="/wholesale-car+vehicles+and+accessories/">Car,Vehicles and Accessories</a></dd><dd><a href="/wholesale-lighting+lamps+and+accessories/">Lighting Lamps and Accessories</a></dd><dd><a href="/wholesale-other+car+vehicles+and+accessories/">Other Car,Vehicles and Accessories</a></dd><dd><a href="/wholesale-parking+sensor/">Parking Sensor</a></dd><dd><a href="/wholesale-protective+gear/">Protective Gear</a></dd></dl> | |
</div> |
@LCM: 太长了 不好意思哈
@miloss: 没事。等下我再看了。下班了。半个小时左右到家。
@LCM: 我也快下班啊 我等你哈
@LCM: 加我Q吧 1054509745 老鸟得帮助菜鸟哈
你帮我匹配第一<dl></dl>标签里面的那些吧 每个dl标签为一类 要分类插入 所以匹配条件是class为lists的div dl标签开头是<dt><a href="/categories/apparel/">Apparel</a></dt>的
@miloss: 你试一下这个,可以匹配全部的。(?<=<div([\w\W]*?)>\s*<dl>([\w\W]*?))([\w\W]*?)<a[^>]*>(?<c>[^<]*)</a>([\w\W]*?)
@miloss: 我匹配用的C#代码是这样的,str是正则表达式,str2是html内容。
string str = textBox1.Text.Trim();
string str2 = textBox2.Text.Trim();
Regex regex = new Regex(str, RegexOptions.None);
MatchCollection matchCollection = regex.Matches(str2);
listBox1.Items.Clear();
for (int i = 0; i < matchCollection.Count; i++)
{
Match match = matchCollection[i];
listBox1.Items.Add(match.Groups["c"].Value);
}
@LCM: 这个里面还匹配出a标签的href
(?<=<div([\w\W]*?)>\s*<dl>([\w\W]*?))([\w\W]*?)<a([^>]*?)href=\"(?<href>[\w\W]*?)\"([^>]*?)>(?<c>[^<]*)</a>([\w\W]*?)
可以这样获取到href和内容
match.Groups["c"].Value+"||"+match.Groups["href"].Value
@LCM: 这匹配出来的事空的啊 我要分别匹配出没个dl里面的
@miloss: 匹配出来不会是空的,你那要是看不到效果的话,等下我发给你源码。匹配出每个dl里面的,我可以改一下,等下发给你。我上午要去开个会。中午帮你弄下。你把邮箱告诉我,我等下发到你邮箱里。
@LCM: chentaolianjie@sina.com 可以加我Q嘛1054509745 我不会经常烦你的
@miloss: 邮件已发送,里面有注释的。公司登录不了QQ的,并且也很少上QQ。
@LCM: 我看见你这没有去标签的 标签是什么去的啊
@miloss: 我没去标签,我只匹配出我想要的内容。
@LCM: for (int i = 0; i < matchCollection.Count; i++)
{
Match match = matchCollection[i];
string name = match.Groups["ass"].Value;
//listBox1.Items.Add(match.Groups["c"].Value);
}
你看我这样获取的单个 单词 怎么就是空的
@miloss: string url = "http://www.tootoomart.com/allcategories/";
string charset = "utf-8";
string html =GetHtmlSource(url, charset); //获取页面html
string pattern_Apparel=@"(?<=<div([\w\W]*?)>\s*<dl>([\w\W]*?))([\w\W]*?)<a[^>]*>(?<c>[^<]*)</a>([\w\W]*?)";
Regex regex = new Regex(pattern_Apparel, RegexOptions.None);
MatchCollection matchCollection = regex.Matches(html);
for (int i = 0; i < matchCollection.Count; i++)
{
Match match = matchCollection[i];
string name = match.Groups["ass"].Value;
//listBox1.Items.Add(match.Groups["c"].Value);
}
我不要连接 只要那 单词
@miloss: 你的这里面没有ass的组啊。网上查一下正则表达式怎么分组的。
@LCM: 我用变量 i 的时候name也为空啊 有时有值 我想把这name 直接先存进数据库
@miloss: 你分组的话你的正则表达式里面要有 (?<组名>正则表达式) 这样的,你才能用match.Groups["ass"].Value读到组内的东西的。ass就是上面定义的组名。否则哪会有内容呢。
写爬虫的童鞋可以试试神箭手云爬虫,自带JS渲染、代理ip、验证码识别等功能,还可以发布和导出爬取的数据,生成图表等,都在云端进行,不需要安装开发环境。