Archives
/
dhtcrawler2
mirror of https://github.com/btdigg-org/dhtcrawler2
2
0
Fork 0
Code Issues Projects Releases Wiki Activity

Commit Graph

  • 45ee021ebb optimize highlight for http master Kevin Lynx 2013-08-24 17:39:44 +0800
  • 82f14c458c optimize highlight for http src Kevin Lynx 2013-08-24 17:39:09 +0800
  • c0b383a7b7 add http search result highlight Kevin Lynx 2013-08-24 16:42:51 +0800
  • b1869dd122 add http search result highlight Kevin Lynx 2013-08-24 16:42:04 +0800
  • 7f7045883e fix stats 1970 bug Kevin Lynx 2013-08-22 21:05:25 +0800
  • 2f8842a18d fix stats 1970 bug Kevin Lynx 2013-08-22 21:04:15 +0800
  • 53e3036f4f add sphinx config doc Kevin Lynx 2013-08-14 20:48:35 +0800
  • ddaa44e6e8 add sphinx config doc Kevin Lynx 2013-08-14 20:47:41 +0800
  • 2da984b5f9 update readme Kevin Lynx 2013-08-12 20:59:33 +0800
  • fa0b80908a update readme Kevin Lynx 2013-08-12 20:58:56 +0800
  • 4a9b85c973 fix sphinx builder query range bug; improve sphinx builder db query performance, to avoid `getmore' command and the 4M useless response Kevin Lynx 2013-08-07 21:50:06 +0800
  • b0428a1e5e fix sphinx builder query range bug; improve sphinx builder db query performance, to avoid `getmore' command and the 4M useless response Kevin Lynx 2013-08-07 21:48:11 +0800
  • f887208cd5 log bug Kevin Lynx 2013-08-06 21:13:24 +0800
  • 4b48c8aed2 log bug Kevin Lynx 2013-08-06 20:57:00 +0800
  • ba64278d12 build date index at startup Kevin Lynx 2013-08-06 20:50:49 +0800
  • 395d73b000 build date index at startup Kevin Lynx 2013-08-06 20:50:08 +0800
  • ac0116a5b3 daman! fix crawler initiali id generation bug (not set random seed) Kevin Lynx 2013-08-05 22:04:37 +0800
  • 648fe7f20d daman! fix crawler initiali id generation bug (not set random seed) Kevin Lynx 2013-08-05 22:03:54 +0800
  • 377912982d add some debug log to sphinx_builder, test sphinx_builder when there's no hashes there and got new hashes Kevin Lynx 2013-08-05 21:31:16 +0800
  • 04a20c177d add some debug log to sphinx_builder, test sphinx_builder when there's no hashes there and got new hashes Kevin Lynx 2013-08-05 21:30:29 +0800
  • 5fa961bc9a add log level config for sphinx_builder Kevin Lynx 2013-08-04 21:42:26 +0800
  • 7af23fcc49 add log level config for sphinx_builder Kevin Lynx 2013-08-04 21:41:46 +0800
  • db83eecfd5 change sphinx_builder, query from mongodb by `skip' really don't work well, build a date index to query by date range Kevin Lynx 2013-08-04 21:37:50 +0800
  • cd1fab1ecb change sphinx_builder, query from mongodb by `skip' really don't work well, build a date index to query by date range Kevin Lynx 2013-08-04 21:34:13 +0800
  • 55ac362bf1 update to the newest kdht, to fix some invalid message error bug Kevin Lynx 2013-08-03 21:39:58 +0800
  • 6f6aac3b35 adjust crawler log directory Kevin Lynx 2013-08-03 21:39:14 +0800
  • 917c222b16 http ui stuff Kevin Lynx 2013-08-03 17:21:16 +0800
  • f16d25dae7 http ui stuff Kevin Lynx 2013-08-03 17:20:52 +0800
  • 60bb12538e http ui adjust Kevin Lynx 2013-08-03 17:10:42 +0800
  • a76f6e5d46 http ui adjust Kevin Lynx 2013-08-03 17:10:15 +0800
  • e8e3142235 modify `giza' library so that i can get sphinx search stats, and because of this, i can add a more detailed page navigation Kevin Lynx 2013-08-03 17:00:26 +0800
  • aed757f2a8 modify `giza' library so that i can get sphinx search stats, and because of this, i can add a more detailed page navigation Kevin Lynx 2013-08-03 16:58:04 +0800
  • 237d90f81a add a new config `search_method', if set to `sphinx', hash reader will not create name_array, also add config for httpd, to config the search method Kevin Lynx 2013-08-03 15:49:29 +0800
  • 44464b40b3 add a new config `search_method', if set to `sphinx', hash reader will not create name_array, also add config for httpd, to config the search method Kevin Lynx 2013-08-03 15:48:03 +0800
  • e816757fef add a new config to head_reader `use_sphinx', when this flag is true, the reader will not create `name_array' for torrent document; now http use sphinx search Kevin Lynx 2013-08-03 14:57:46 +0800
  • 15f398023c log disable Kevin Lynx 2013-08-02 22:20:29 +0800
  • 486c354ba0 change sphinx torrent loading using an existing cursor Kevin Lynx 2013-08-02 22:19:31 +0800
  • 6b1c0a0a26 change sphinx torrent loading using an existing cursor Kevin Lynx 2013-08-02 22:18:07 +0800
  • bc00e03b33 fix sphinx xml utf8 related issure, filter these unicode control characters, only backup delta file if the operation failed Kevin Lynx 2013-08-01 23:20:28 +0800
  • 92826bf848 turn off the damn debug log Kevin Lynx 2013-08-01 23:20:02 +0800
  • 79291ab4e9 fix sphinx xml utf8 related issure, filter these unicode control characters, only backup delta file if the operation failed Kevin Lynx 2013-08-01 23:17:52 +0800
  • 1d870e2e42 add sphinx search stats Kevin Lynx 2013-07-31 22:06:18 +0800
  • 5e9c36f787 add sphinx search stats Kevin Lynx 2013-07-31 22:05:53 +0800
  • 0bdac737ad add a simple page navigation for sphinx_search Kevin Lynx 2013-07-31 20:57:35 +0800
  • 1d27f2416b add a simple page navigation for sphinx_search Kevin Lynx 2013-07-31 20:56:48 +0800
  • 40f2bae9b8 fix sphinx_build memory leak bug, caused by mongo_cursor Kevin Lynx 2013-07-31 12:17:21 +0800
  • e1c905b0a7 fix sphinx_build memory leak bug, caused by mongo_cursor Kevin Lynx 2013-07-31 12:16:14 +0800
  • 46c99cabd8 sphinx worker call infinity Kevin Lynx 2013-07-30 22:43:32 +0800
  • 149f10724e sphinx worker call infinity Kevin Lynx 2013-07-30 22:43:02 +0800
  • 18edffc2a1 fix some sphinx related bugs, now it can be used to build sphinx index, still in experiment stage, add `giza' library to query sphinx in http_fontend Kevin Lynx 2013-07-30 22:17:31 +0800
  • 7b1a435a43 fix some sphinx related bugs, now it can be used to build sphinx index, still in experiment stage, add `giza' library to query sphinx in http_fontend Kevin Lynx 2013-07-30 22:14:28 +0800
  • 7ab79b5d2e variable name change Kevin Lynx 2013-07-29 23:26:12 +0800
  • e5011ab75a fix sphinx doc creation failed Kevin Lynx 2013-07-29 23:14:41 +0800
  • f242d4e44f add sphinx support, in expirment status right now Kevin Lynx 2013-07-29 23:03:39 +0800
  • 60472bd731 add LICENSE.txt, lincensed by MIT Kevin Lynx 2013-07-24 20:13:49 +0800
  • b961dc9c46 add LICENSE.txt, lincensed by MIT Kevin Lynx 2013-07-24 20:13:12 +0800
  • 0c67e46e5c fix daterange issure which not only record today torrents, not it only show the today inserted torrents Kevin Lynx 2013-07-23 22:16:40 +0800
  • ec456de63d fix daterange issure which not only record today torrents, not it only show the today inserted torrents Kevin Lynx 2013-07-23 22:15:08 +0800
  • 4dc05bf2cc adjust http stats display Kevin Lynx 2013-07-23 21:45:31 +0800
  • 28acbdaa45 adjust http stats display Kevin Lynx 2013-07-23 21:45:06 +0800
  • 94a2ac34bc system stats adjust, add more stats to http front-end Kevin Lynx 2013-07-23 21:41:08 +0800
  • cb914fe609 system stats adjust, add more stats to http front-end Kevin Lynx 2013-07-23 21:40:17 +0800
  • 2a9f99940a add a new force to string log func, add log to httpd, it can log unicode characters to logfiles Kevin Lynx 2013-07-22 22:59:10 +0800
  • 6fbd0cb218 add a new force to string log func, add log to httpd, it can log unicode characters to logfiles Kevin Lynx 2013-07-22 22:58:07 +0800
  • 3b0e5701c8 complete all http uri to json api Kevin Lynx 2013-07-22 21:24:56 +0800
  • 928798ed28 complete all http uri to json api Kevin Lynx 2013-07-22 21:23:44 +0800
  • 980c6cad57 add query stats for new hash_writer Kevin Lynx 2013-07-21 22:20:47 +0800
  • 13d35a44c1 add query stats for new hash_writer Kevin Lynx 2013-07-21 22:20:16 +0800
  • 070e97e826 add hash filter stats to the new hash_reader Kevin Lynx 2013-07-21 22:10:05 +0800
  • e46c264056 add `size' function to hash_download_cache, to debug Kevin Lynx 2013-07-21 21:55:30 +0800
  • 5d211c3f14 add `size' function to hash_download_cache, to debug Kevin Lynx 2013-07-21 21:52:44 +0800
  • 3864940905 fix hash_download startup bug Kevin Lynx 2013-07-21 21:32:19 +0800
  • 67ff84adaa fix hash_download_cache startup bug Kevin Lynx 2013-07-21 21:30:28 +0800
  • 108a1bfd1b fix hash_download_cache startup bug Kevin Lynx 2013-07-21 21:29:47 +0800
  • dcf0181839 NOTE: rewrite hash_reader, config changed, dht_hash database changed, require to remove existed dht_hash database Kevin Lynx 2013-07-21 21:18:40 +0800
  • e5b35e58ed NOTE: rewrite hash_reader, config changed, dht_hash database changed, require to remove existed dht_hash database Kevin Lynx 2013-07-21 21:13:05 +0800
  • 72c35be437 change default config Kevin Lynx 2013-07-21 09:24:33 +0800
  • 75b3d82f4c change default config Kevin Lynx 2013-07-21 09:23:48 +0800
  • 060804ae31 fix cache_indexer bug Kevin Lynx 2013-07-20 19:38:16 +0800
  • d00c84135b fix cache_indexer message leak bug Kevin Lynx 2013-07-20 19:37:41 +0800
  • d9deb8dfc9 add simple `get' json api, fix http search space decode Kevin Lynx 2013-07-20 10:57:27 +0800
  • 2658040f3a add simple `get' json api, fix http search space decode Kevin Lynx 2013-07-20 10:56:41 +0800
  • 54a30122fa fix hash_date Kevin Lynx 2013-07-19 21:32:10 +0800
  • ba92e9cd77 fix hash_date Kevin Lynx 2013-07-19 21:31:36 +0800
  • 37ccb19575 change hash_date only record the new inserted torrents Kevin Lynx 2013-07-19 21:08:32 +0800
  • 28fe69d141 hash_date only record today new inserted torrents Kevin Lynx 2013-07-19 21:00:37 +0800
  • 76542be37a config max download task per hash-reader Kevin Lynx 2013-07-18 22:04:57 +0800
  • 45ca7d584e config max download task per hash-reader, Kevin Lynx 2013-07-18 22:03:47 +0800
  • 35a131fa8f nothing Kevin Lynx 2013-07-18 14:03:34 +0800
  • 4882cbf692 fix hash-writer cache-writting issure Kevin Lynx 2013-07-18 13:59:37 +0800
  • 976740ea57 hash_writer write cache hashes 100 by 100, not all caches Kevin Lynx 2013-07-18 13:56:51 +0800
  • 928fc86934 recompile Kevin Lynx 2013-07-18 13:17:06 +0800
  • cd9ae2ec53 Merge branch 'src' of github.com:kevinlynx/dhtcrawler2 into src Kevin Lynx 2013-07-18 13:09:12 +0800
  • 5592b3989b fix hash_reader stop working bug Kevin Lynx 2013-07-18 12:43:51 +0800
  • f5655ba0f3 fix hash_reader stop working bug Kevin Lynx 2013-07-18 12:38:31 +0800
  • 01451534ee change crawler to cache hashes and merge hashes before inserted into db Kevin Lynx 2013-07-17 23:32:30 +0800
  • 810464330d NOTE: big change! Need to delete config files. The crawler will cache hashes and merge duplicated queries. Kevin Lynx 2013-07-17 22:55:35 +0800
  • 629e92115d fix cache_indexer download bug Kevin Lynx 2013-07-17 19:11:01 +0800
  • 1dc4a2c588 fix cache_indexer download error Kevin Lynx 2013-07-17 17:44:28 +0800
  • ff338f2c9b fix cache_indexer state not saved correctly Kevin Lynx 2013-07-16 22:49:08 +0800