BioHackersNet|
Size: 13706
Comment:
|
Size: 2515
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| !'''[http://generic-viagra.land.ru/ | Generic viagra online] !''' !'''[http://cheap-phentermine.hotmail.ru/ | buy phentermine cheap phentermine online] !''' !'''[http://best-phentermine.hotmail.ru/ | buy cheap phentermine online] !''' !'''[http://fha_mortgage.chat.ru/ | buy cheap phentermine online] !''' !'''[http://ticket1_broker.chat.ru/ | Ticket Broker] !''' !'''[http://buy_phent.chat.ru/ | buy adipex phentermine] !''' !'''[http://cheap--viagra.fromru.com/ | Cheap viagra] !'''! |
웹서버([HTTP])를 순회하며 각 홈페이지에 있는 수많은 정보를 수집하는 프로그램으로, 사람이 일일이 홈페이지의 각 링크를 따라가서 정보를 얻는 반복적인 작업을 대신하여 프로그램이 스스로 데이터를 분석하고 그 안의 [URL]들을 추출하여 다른 [URL]로 이동하면서 정보를 가지고 오는 것이다. 웹의 여기저기를 돌아다닌다는 의미에서 WWW wanderer, 웹이 그물 또는 거미줄을 의미하는 점에서 WWW spider라 불리기도 한다. |
| Line 9: | Line 3: |
| !'''[http://united-airline.boom.ru/ | United Airlines tickets flights] !'''! !'''[http://computer-desk.fromru.com/ | Computer desk] !'''! |
주로 다음의 것들이 있다. |
| Line 12: | Line 5: |
| [http://online1degree.chat.ru/ | Online degree] [http://camel-cig.newmail.ru | Cheap camel cigarettes] [http://marlb-cigarette.newmail.ru | marlboro cigarettes] [http://gift_basket.chat.ru/ | Gift basket] [http://find-tires.com/ | Best Tires] [http://craps_internet.chat.ru/ | craps internet] [http://craps-internet.pochta.ru/ | craps internet] [http://craps-internet.land.ru/ | craps internet] [http://casino_craps_onl.chat.ru/ | casino craps online] [http://casino-craps-online.pochta.ru/ | casino craps online] [http://casino-craps-online.fromru.com/ | casino craps online] [http://craps_playing.chat.ru/ | craps playing] [http://craps-playing.pochta.ru/ | craps playing] [http://craps-playing.land.ru/ | craps playing] [http://online-craps.fromru.com/ | online craps] [http://craps_online1.chat.ru/ | online craps] |
|| '''주요기능''' || '''WebRobot''' || || 자원발견 DB구축 || [Google]bot 등 검색엔진용 || || 연결점확인 || MOM spider, EIT Link Verifier Robot || || Web site mirroring || HTMLgobble, GetUrl, WebCopy, WebFetcher || || 웹성장 모니터링 || W4 || |
| Line 29: | Line 11: |
| 웹로봇이 멍청하다는 문제점이 있다. 특히 많은 부분 고려를 안하고 대충만들어지게 되면, 웹서버를 죽이던가 하여 다른 시스템들을 파괴할 수 있다. 이를 막기 위해, robot.txt 파일을 사용한다. 거의 무시당하는 경우가 많은듯. 도저히 안되면 IpToBlock방법밖에 없다. | |
| Line 30: | Line 13: |
| Best links: '''credit card''' | robot.txt 파일의 형식은 SeeAlso [http://www.robotstxt.org/wc/norobots.html robot.txt 표준] {{{ # --> 주석문 User-agent: --> Agent의 이름, 최소한 '*'라도 입력 Disallow: --> Robot이 방문할 수 없는 URL 기술, 전체 경로나 부분 경로 설정 }}} |
| Line 32: | Line 20: |
| [http://accept_cred_card.chat.ru/ | accept credit card] [http://accept-credit-card.fromru.com/ | accept credit card] |
[Perky]씨의 크롤러 필터링내용 in httpd.conf --[yong27], 2004-04-26 {{{ BrowserMatch "HTTrack" bad_bot BrowserMatch "Download Ninja 2.0" bad_bot BrowserMatch "JBH Agent 2.0" bad_bot BrowserMatch "EmailCollector/1.0" spam_bot BrowserMatch "EmailSiphon" spam_bot BrowserMatch "EmailWolf 1.00" spam_bot BrowserMatch "ExtractorPro" spam_bot BrowserMatch "Crescent Internet ToolPak HTTP OLE Control v.1.0" spam_bot BrowserMatch "Mozilla/2.0 (compatible; NEWT ActiveX; Win32)" spam_bot BrowserMatch "CherryPicker/1.0" spam_bot BrowserMatch "CherryPickerSE/1.0" spam_bot BrowserMatch "CherryPickerElite/1.0" spam_bot BrowserMatch "NICErsPRO" spam_bot BrowserMatch "WebBandit/2.1" spam_bot BrowserMatch "WebBandit/3.50" spam_bot BrowserMatch "webbandit/4.00.0" spam_bot BrowserMatch "WebEMailExtractor/1.0B" spam_bot BrowserMatch "autoemailspider" spam_bot }}} |
| Line 35: | Line 42: |
| 주요로봇현황 || [Google]bot || 64.68.*.* || |
|
| Line 36: | Line 45: |
| Best links: '''credit counseling''' [http://credit1counselin.chat.ru/ | credit counseling] Best links: '''credit help''' [http://credit_help.chat.ru/ | credit help] Best links: '''credit-report''' [http://credit_report.chat.ru/ | credit-report] Best links: '''cruise''' [http://norwegiancruisel.chat.ru/ | norwegian cruise line] [http://carnivalcruise.chat.ru/ | carnival cruise] [http://caribbeancruise.chat.ru/ | caribbean cruise] [http://pcruiseline.chat.ru/ | princess cruise line] [http://cruisetravel.chat.ru/ | cruise travel] [http://crystal1cruise.chat.ru/ | crystal cruise] [http://costacruises.chat.ru/ | costa cruises] [http://radissoncruise.chat.ru/ | radisson cruise] [http://medit1cruise.chat.ru/ | mediterranean cruise] [http://panama1cruise.chat.ru/ | panama canal cruise] [http://cruisedeal.chat.ru/ | cruise deal] [http://cruiseprincess.chat.ru/ | cruise princess] [http://norwegiancruise.chat.ru/ | norwegian cruise] [http://cruiseline.chat.ru/ | cruise line] [http://cruiseship.chat.ru/ | cruise ship] [http://discountcruises.chat.ru/ | discount cruises] [http://honeymooncruise.chat.ru/ | honeymoon cruise] [http://cruiseeuropean.chat.ru/ | cruise european] [http://celebritycruise.chat.ru/ | celebrity cruise] [http://lastminutecruise.chat.ru/ | last minute cruise] [http://cruisespecial.chat.ru/ | cruise special] [http://alaskacruise.chat.ru/ | alaska cruise] [http://carnivalcruises.chat.ru/ | carnival cruise ship] [http://newyorkcruises.chat.ru/ | new york cruises] [http://cheapcruise.chat.ru/ | cheap cruise] [http://hawaiicruises.chat.ru/ | hawaii cruise] [http://cruisevacation.chat.ru/ | cruise vacation] [http://disneycruise.chat.ru/ | disney cruise] [http://hamericacruises.chat.ru/ | holland america cruises] [http://familycruises.chat.ru/ | family cruises] [http://bahamascruise.chat.ru/ | bahamas cruise] [http://mexicocruise.chat.ru/ | mexico cruise] [http://bermudacruise.chat.ru/ | bermuda cruise] [http://europecruise.chat.ru/ | europe cruise] [http://caribbeancruisel.chat.ru/ | caribbean cruise line] [http://cruisepackage.chat.ru/ | cruise package] [http://cruisereview.chat.ru/ | cruise review] [http://carnivalcruisel.chat.ru/ | carnival cruise line] [http://disneycruisel.chat.ru/ | disney cruise line] [http://nilecruises.chat.ru/ | nile cruises] [http://luxurycruise.chat.ru/ | luxury cruise] [http://neworleanscruise.chat.ru/ | new orleans cruises] [http://celebritycruisel.chat.ru/ | celebrity cruise line] [http://galvestoncruises.chat.ru/ | galveston cruises] [http://starcruise.chat.ru/ | star cruise] [http://cruiseholiday.chat.ru/ | cruise holiday] [http://honeymooncruise.boom.ru/ | honeymoon cruise] [http://caribbeancruisel.boom.ru/ | caribbean cruise line] [http://ccruise-ship.boom.ru/ | carnival cruise ship] [http://europe-cruise.boom.ru/ | europe cruise] [http://cruise-package.boom.ru/ | cruise package] [http://cruise-european.boom.ru/ | cruise european] [http://bermuda-cruise.boom.ru/ | bermuda cruise] [http://bahamas-cruise.boom.ru/ | bahamas cruise] [http://mexico-cruise.boom.ru/ | mexico cruise] [http://cruise-review.boom.ru/ | cruise review] [http://cruise-special.boom.ru/ | cruise special] [http://disney-cruisel.boom.ru/ | disney cruise line] [http://norwegian-cruise.boom.ru/ | norwegian cruise] [http://lminute-cruise.boom.ru/ | last minute cruise] [http://italy-cruise.boom.ru/ | italy cruise] [http://p-cruise-line.boom.ru/ | princess cruise line] [http://hawaii-cruise.boom.ru/ | hawaii cruise] [http://mediterranean.boom.ru/ | mediterranean cruise] [http://carnival-cruisel.boom.ru/ | carnival cruise line] [http://luxury-cruise.boom.ru/ | luxury cruise] [http://cruise-deal.boom.ru/ | cruise deal] [http://cruise-line.boom.ru/ | cruise line] [http://n-cruise-line.boom.ru/ | norwegian cruise line] [http://carnival-cruise.boom.ru/ | carnival cruise] [http://caribbeancruise.boom.ru/ | caribbean cruise] [http://crystal--cruise.boom.ru/ | crystal cruise] [http://cruise1.boom.ru/ | cruise] [http://cruise1.chat.ru/ | cruise] [http://cruise_ship.chat.ru/ | cruise ship] [http://cruise-ship.fromru.com/ | cruise ship] [http://luxury_cruise.chat.ru/ | luxury cruise] [http://luxury-cruise.fromru.com/ | luxury cruise] [http://caribbean_cruise.chat.ru/ | caribbean cruise] [http://caribbean-cruise.pisem.net/ | caribbean cruise] [http://haw_cruise.chat.ru/ | hawaii cruise] [http://hawaii1-cruise.fromru.com/ | hawaii cruise] Best links: '''cruise line''' [http://cruiseline.boom.ru/ | cruise line] Best links: '''cruise review''' [http://cruise_review.euro.ru/ | cruise review] Best links: '''cruises''' [http://crystal-cruise.boom.ru/ | crystal cruise] [http://cruise2crystal.chat.ru/ | crystal cruise] [http://honeymoon-crs.boom.ru/ | honeymoon cruise] [http://honeymoon1cruise.chat.ru/ | honeymoon cruise] [http://panama-cruise.boom.ru/ | panama canal cruise] [http://cruise2panama.chat.ru/ | panama cruises] [http://cruise2caribbean.chat.ru/ | caribbean cruise] [http://hawaiian-cruise.boom.ru/ | hawaiian cruise] [http://hawaiian1cruise.chat.ru/ | hawaii cruise] [http://celebrcruiseline.chat.ru/ | celebrity cruise line] [http://celebrcruiseline.boom.ru/ | celebrity cruise line] [http://cruise1line.chat.ru/ | cruise line] [http://hawaiicruise.boom.ru/ | hawaii cruise] [http://hawaiicruise.chat.ru/ | hawaii cruise] [http://radisson-cruise.boom.ru/ | radisson cruise] [http://radisson4cruise.chat.ru/ | radisson cruise] [http://norwegcruiselin.boom.ru/ | norwegian cruise line] [http://luxury2cruise.boom.ru/ | luxury cruise] [http://luxury2cruise.chat.ru/ | luxury cruise] [http://disney2cruise.chat.ru/ | disney cruise] [http://discount2cruises.chat.ru/ | discount cruises] [http://cruise2travel.chat.ru/ | cruise travel] [http://cruise2ship.chat.ru/ | cruise ship] [http://cruisedeal.boom.ru/ | cruise deal] [http://cruise1deal.chat.ru/ | cruise deal] [http://caribbean2cruise.boom.ru/ | caribbean cruise] [http://caribbean2cruise.chat.ru/ | caribbean cruise] [http://cruise4vacation.chat.ru/ | cruise vacation] [http://asia-cruise.boom.ru/ | asia cruise] [http://asiacruise.chat.ru/ | asia cruise] [http://princescruiselin.boom.ru/ | princess cruise line] [http://princescruiselin.chat.ru/ | princess cruise line] [http://italy2cruise.boom.ru/ | italy cruise] [http://tom-cruise.boom.ru/ | tom cruise] [http://tom4cruise.chat.ru/ | tom cruise] [http://bermuda2cruise.boom.ru/ | bermuda cruise] [http://bermuda4cruise.chat.ru/ | bermuda cruise] [http://disneycruiseline.boom.ru/ | disney cruise line] [http://disneycruiseline.chat.ru/ | disney cruise line] [http://lastminutcruises.chat.ru/ | last minute cruises] [http://european-cruise.boom.ru/ | european cruise] [http://europeancruise.chat.ru/ | european cruise] [http://norwegian2cruise.boom.ru/ | norwegian cruise] [http://norwegian4cruise.chat.ru/ | norwegian cruise] [http://barge-cruise.boom.ru/ | barge cruise] [http://bargecruise.chat.ru/ | barge cruise] [http://tropical-cruise.boom.ru/ | tropical cruise] [http://mexico2-cruise.boom.ru/ | mexico cruise] [http://canada-cruise.boom.ru/ | canada cruise] [http://canada2cruise.chat.ru/ | canada cruise] [http://cruise4special.boom.ru/ | cruise special] [http://cruise4special.chat.ru/ | cruise special] [http://single4cruises.chat.ru/ | single cruises] [http://cruise4-review.boom.ru/ | cruise review] [http://cruise2review.chat.ru/ | cruise review] [http://cruise2bahamas.chat.ru/ | bahamas cruise] [http://nile2cruises.chat.ru/ | nile cruises] Best links: '''crystal cruise''' [http://crystalcruise.chat.ru/ | crystal cruise] [http://crystal1cruise.boom.ru/ | crystal cruise] Best links: '''debt help self''' [http://debt_help_self.chat.ru/ | debt help self] Best links: '''decorating''' [http://interior-decorating.fromru.com/ | interior decorating] [http://interior_decorat.chat.ru/ | interior decorating] [http://interior-decorating.pochta.ru/ | interior decorating] [http://interior_deco.chat.ru/ | interior decorating] [http://interior-deco.pochta.ru/ | interior decorating] Best links: '''delivery flowers''' [http://delivery_flowers.chat.ru/ | delivery flower] Best links: '''dell laptop used''' [http://dell_laptop_used.chat.ru/ | dell laptop used] Best links: '''dental''' [http://dental_schools.chat.ru/ | dental schools] Best links: '''dental care''' [http://dental-care.fromru.com/ | dental care] Best links: '''diabetic''' [http://diabetic-supply.fromru.com/ | diabetic supply] [http://diabetic_supply.chat.ru/ | diabetic supply] [http://diabetic2_supply.chat.ru/ | diabetic supply] [http://diabetic-supply.pochta.ru/ | diabetic supply] Best links: '''didrex''' [http://didrex1.chat.ru/ | didrex] Best links: '''digital photo''' [http://digital_photo.euro.ru/ | digital photo] Best links: '''diploma''' [http://high-school-diploma.fromru.com/ | high school diploma] [http://fake-diploma-.pochta.ru/ | fake diploma] [http://high_sch_diploma.chat.ru/ | high school diploma] [http://fake1_diploma.chat.ru/ | fake diploma] Best links: '''discount computer''' [http://discountcomputer.euro.ru/ | discount computer] Best links: '''discount cruises''' [http://discount_cruises.euro.ru/ | discount cruise] Best links: '''discount hotels''' [http://discount-hotels.fromru.com/ | discount hotel] Best links: '''divorce''' [http://floridadivorce.chat.ru/ | divorce florida] [http://ncarolinadivorce.chat.ru/ | carolina divorce north] [http://divorce1help.chat.ru/ | divorce help] [http://divorce1form.chat.ru/ | divorce form] [http://divorce1advice.chat.ru/ | advice divorce] [http://nydivorce.chat.ru/ | divorce new york] [http://divorceuncont.chat.ru/ | divorce uncontested] Best links: '''dog''' [http://dog-breed.pochta.ru/ | dog breed] [http://dog_health.chat.ru/ | dog health] [http://dog-health.pochta.ru/ | dog health] [http://dog_collar.chat.ru/ | dog collar] [http://dog-collar.pochta.ru/ | dog collar] [http://dog_grooming.chat.ru/ | dog grooming] [http://dog-grooming.pochta.ru/ | dog grooming] [http://dog_bone.chat.ru/ | dog bone] [http://dog-bone.pochta.ru/ | dog bone] [http://dog1-house.chat.ru/ | dog house] [http://dog-house.pochta.ru/ | dog house] [http://dog_t_shirt.chat.ru/ | dog t shirt] [http://dog-t-shirt.pochta.ru/ | dog t shirt] Best links: '''dog bed''' [http://dog_bed.chat.ru/ | dog bed] [http://dog-bed.pochta.ru/ | dog bed] Best links: '''dog breed''' [http://dog1_breed.chat.ru/ | dog breed] Best links: '''dog care''' [http://dog_care.chat.ru/ | dog care] [http://dog-care.pochta.ru/ | dog care] Best links: '''dog food''' [http://dog_food.chat.ru/ | dog food] [http://dog-food.pochta.ru/ | dog food] Best links: '''dog gift''' [http://dog_gift.chat.ru/ | dog gift] [http://dog-gift.pochta.ru/ | dog gift] Best links: '''dog pile''' [http://dog_pile.chat.ru/ | dog pile] [http://dog-pile.pochta.ru/ | dog pile] Best links: '''dog supply''' [http://dog_supply.chat.ru/ | dog supply] [http://dog-supply.pochta.ru/ | dog supply] Best links: '''dog toy''' [http://dog_toy.chat.ru/ | dog toy] [http://dog-toy.pochta.ru/ | dog toy] Best links: '''dog training''' [http://dog_training.chat.ru/ | dog training] [http://dog-training.pochta.ru/ | dog training] Best links: '''dog treat''' [http://dog_treat.chat.ru/ | dog treat] [http://sympathy-giftb.fromru.com/ | sympathy gift basket] [http://baby-gb.pochta.ru/ | baby gift basket] [http://food_gb.chat.ru/ | food gift basket] [http://gourmet_gb.chat.ru/ | gourmet gift basket] [http://cookie_gift_b.chat.ru/ | cookie gift basket] |
관련자료 * http://user.chollian.net/~jolra/study/robotagent.html |
웹서버([HTTP])를 순회하며 각 홈페이지에 있는 수많은 정보를 수집하는 프로그램으로, 사람이 일일이 홈페이지의 각 링크를 따라가서 정보를 얻는 반복적인 작업을 대신하여 프로그램이 스스로 데이터를 분석하고 그 안의 [URL]들을 추출하여 다른 [URL]로 이동하면서 정보를 가지고 오는 것이다. 웹의 여기저기를 돌아다닌다는 의미에서 WWW wanderer, 웹이 그물 또는 거미줄을 의미하는 점에서 WWW spider라 불리기도 한다.
주로 다음의 것들이 있다.
주요기능 |
WebRobot |
자원발견 DB구축 |
[Google]bot 등 검색엔진용 |
연결점확인 |
MOM spider, EIT Link Verifier Robot |
Web site mirroring |
HTMLgobble, GetUrl, WebCopy, WebFetcher |
웹성장 모니터링 |
W4 |
웹로봇이 멍청하다는 문제점이 있다. 특히 많은 부분 고려를 안하고 대충만들어지게 되면, 웹서버를 죽이던가 하여 다른 시스템들을 파괴할 수 있다. 이를 막기 위해, robot.txt 파일을 사용한다. 거의 무시당하는 경우가 많은듯. 도저히 안되면 IpToBlock방법밖에 없다.
robot.txt 파일의 형식은 SeeAlso [http://www.robotstxt.org/wc/norobots.html robot.txt 표준]
# --> 주석문 User-agent: --> Agent의 이름, 최소한 '*'라도 입력 Disallow: --> Robot이 방문할 수 없는 URL 기술, 전체 경로나 부분 경로 설정
[Perky]씨의 크롤러 필터링내용 in httpd.conf --[yong27], 2004-04-26
BrowserMatch "HTTrack" bad_bot BrowserMatch "Download Ninja 2.0" bad_bot BrowserMatch "JBH Agent 2.0" bad_bot BrowserMatch "EmailCollector/1.0" spam_bot BrowserMatch "EmailSiphon" spam_bot BrowserMatch "EmailWolf 1.00" spam_bot BrowserMatch "ExtractorPro" spam_bot BrowserMatch "Crescent Internet ToolPak HTTP OLE Control v.1.0" spam_bot BrowserMatch "Mozilla/2.0 (compatible; NEWT ActiveX; Win32)" spam_bot BrowserMatch "CherryPicker/1.0" spam_bot BrowserMatch "CherryPickerSE/1.0" spam_bot BrowserMatch "CherryPickerElite/1.0" spam_bot BrowserMatch "NICErsPRO" spam_bot BrowserMatch "WebBandit/2.1" spam_bot BrowserMatch "WebBandit/3.50" spam_bot BrowserMatch "webbandit/4.00.0" spam_bot BrowserMatch "WebEMailExtractor/1.0B" spam_bot BrowserMatch "autoemailspider" spam_bot
주요로봇현황
[Google]bot |
64.68.*.* |
관련자료