原文地址:C#中HttpClient使用注意:预热与长连接 最近在测试一个第三方API,准备集成在我们的网站应用中。API的调用使用的是.NET中的HttpClient,由于这个API会在关键业务中用到,对调用API的整体响应速度有严格要求,所以对HttpClient有了格外的关注。 开始测试的时候,只在客户端通过HttpClient用PostAsync发了一个http post请求。测试时发现,从创建HttpClient实例,到发出请求,到读取到服务器的响应数据总耗时在2s左右,而且多次测试都是这样。2s的响应速度当然是无法让人接受的,我们希望至少控制在100ms以内。于是开始追查这个问题的原因。 在API的返回数据中包含了该请求在服务端执行的耗时,这个耗时都在20ms以内,问题与服务端API无关。于是把怀疑点放到了网络延迟上,但ping服务器的响应时间都在10ms左右,网络延迟的可能性也不大。 当我们正准备换一个网络环境进行测试时,突然想到,我们的测试方式有些问题。我们只通过HttpClient发了一个PostAsync请求,假如HttpClient在第一次调用时存在某种预热机制(比如在EF中就有这样的机制),现在2s的总耗时可能大多消耗在HttpClient的预热上。 于是修改测试代码,将调用由1次改为100次,然后恍然大悟地发现——只有第1次是2s,接下来的99次都在100ms以内。果然是HttpClient的某种预热机制在搞鬼! 既然知道了是HttpClient预热机制的原因,那我们可以帮HttpClient进行热身,减少第一次请求的耗时。我们尝试了一种预热方式,在正式发http post请求之前,先发一个http head请求,代码如下:
1 2 3 4 |
_httpClient.SendAsync(new HttpRequestMessage { Method = new HttpMethod("HEAD"), RequestUri = new Uri(BASE_ADDRESS + "/") }) .Result.EnsureSuccessStatusCode(); |
经测试,通过这种热身方法,可以将第一次请求的耗时由2s左右降到1s以内(测试结果是700多ms)。 在知道第1次HttpClient请求耗时2s的真相之后,我们将目光转向了剩下的99次耗时100ms以内的请求,发现绝大部分请求都在50ms以上。有没有可能将之降至50ms以下?而且,之前一直有这样的纠结:每次调用是不是一定要对HttpClient进行Dispose()?是不是要将HttpClient单例或者静态化(声明为静态变量)?借此机会一起研究一下。 在HttpClient的背后,有一个对请求响应速度有着不容忽视影响的东东——TCP连接。一个HttpClient实例会关联一个TCP连接,在对HttpClient进行Dispose时,会关闭TCP连接(我们用Wireshark进行网络抓包也验证了这一点)。 在之前的测试中,我们每次用HttpClient发请求时,都是新建一个HttpClient实例,用完就对它进行Dispose,代码如下:
1 2 3 4 |
using (var httpClient = new HttpClient() { BaseAddress = new Uri(BASE_ADDRESS) }) { httpClient.PostAsync("/", new FormUrlEncodedContent(parameters)); } |
所以每次请求时都要经历新建TCP连接->传数据->关闭连接(也就是通常所说的短连接),而且雪上加霜的是请求用的是https,建立TCP连接时还需要一个基于公私钥加解密的key exchange过程:Client Hello -> Server Hello -> Certificate -> Client Key Exchange -> New Session Ticket。 如果我们想将请求响应时间降至50ms以下,就必须从这个地方下手——重用TCP连接(也就是通常所说的长连接)。要实现长连接,首先需要的就是在HttpClient第1次请求后不关闭TCP连接(不调用Dispose方法);而要让后续的请求继续使用这个未关闭的TCP连接,我们必须要使用同一个HttpClient实例;而要使用同一个HttpClient实例,就得实现HttpClient的单例或者静态化。之前的3 个问题,由于要解决第1个问题,后2个问题变成了别无选择。 为了实现长连接,我们将HttpClient的调用代码改为如下的样子:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
public class HttpClientTest { private static readonly HttpClient _httpClient; static HttpClientTest() { _httpClient = new HttpClient() { BaseAddress = new Uri(BASE_ADDRESS) }; //帮HttpClient热身 _httpClient.SendAsync(new HttpRequestMessage { Method = new HttpMethod("HEAD"), RequestUri = new Uri(BASE_ADDRESS + "/") }) .Result.EnsureSuccessStatusCode(); } public async Task<string> PostAsync() { var response = await _httpClient.PostAsync("/", new FormUrlEncodedContent(parameters)); return await response.Content.ReadAsStringAsync(); } } |
然后测试一下请求响应时间:
1 2 3 4 5 6 7 8 9 10 11 |
Elapsed:750ms Elapsed:31ms Elapsed:30ms Elapsed:43ms Elapsed:27ms Elapsed:29ms Elapsed:28ms Elapsed:35ms Elapsed:36ms Elapsed:31ms .... |
除了第1次请求,接下来的99次请求绝大多数都在50ms以内。TCP长连接的效果必须的! 通过Wireshak抓包也验证了长连接的效果: 这时,你也许会产生这样的疑问:将HttpClient声明为静态变量,会不会存在线程安全问题?我们当时也有这样的疑问,后来在stackoverflow上找到了答案:
1 2 3 4 5 6 7 8 9 10 |
As per the comments below (thanks @ischell), the following instance methods are thread safe (all async): CancelPendingRequests DeleteAsync GetAsync GetByteArrayAsync GetStreamAsync GetStringAsync PostAsync PutAsync SendAsync |
HttpClient的所有异步方法都是线程安全的,放心使用。 到这里,HttpClient的问题是不是可以完美收官了?。。。稍等,还有一个问题。 客户端虽然保持着TCP连接,但TCP连接是两口子的事,服务器端呢?你不告诉服务器,服务器怎么知道你要一直保持TCP连接呢?对于客户端,保持TCP连接的开销不大;但是对于服务器,则完全不一样的,如果默认都保持TCP连接,那可是要保持成千上万客户端的连接啊。所以,一般的Web服务器都会根据客户端的诉求来决定是否保持TCP连接,这就是keep-alive存在的理由。 所以,我们还要给HttpClient增加一个Connection:keep-alive的请求头,代码如下:
1 |
_httpClient.DefaultRequestHeaders.Connection.Add("keep-alive"); |
现在终于可以收官了。但是肯定不完美,分享的只是解决问题的过程。 from:https://www.cnblogs.com/JustYong/p/5872296.html
View DetailsGithub只允许上传最大100MB的文件,如果超过,则会被server reject 则需: git filter-branch --force --index-filter "git rm --cached --ignore-unmatch FILEPATH" --prune-empty --tag-name-filter VERSION — --all git commit --amend -CHEAD git push origin master 注意要在.git文件夹目录下执行以上命令 http://www.walbrix.com/jp/blog/2013-10-github-large-files.html from:https://blog.csdn.net/fightforyourdream/article/details/25357121
View Details
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 |
using Newtonsoft.Json; using System.Net; using System.Net.Http; using System.Net.Http.Headers; using System.Threading.Tasks; namespace w3cnet.Utils { /// <summary> /// HttpClient工具类 /// </summary> public class HttpClientUtil { /// <summary> /// GET /// </summary> /// <param name="url"></param> /// <param name="statusCode"></param> /// <returns></returns> public static string Get(string url, out string statusCode) { if (url.StartsWith("https")) ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12; var httpClient = new HttpClient(); httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json")); var response = httpClient.GetAsync(url).Result; statusCode = response.StatusCode.ToString(); if (response.IsSuccessStatusCode) { string result = response.Content.ReadAsStringAsync().Result; return result; } return string.Empty; } /// <summary> /// GET /// </summary> /// <typeparam name="T"></typeparam> /// <param name="url"></param> /// <returns>指定对象</returns> public static T Get<T>(string url) where T : class, new() { if (url.StartsWith("https")) ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12; var httpClient = new HttpClient(); httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json")); var response = httpClient.GetAsync(url).Result; T result = default(T); if (response.IsSuccessStatusCode) { var t = response.Content.ReadAsStringAsync(); var s = t.Result; result = JsonConvert.DeserializeObject<T>(s); } return result; } /// <summary> /// POST /// </summary> /// <param name="url"></param> /// <param name="postData"></param> /// <param name="statusCode"></param> /// <returns></returns> public static string Post(string url, string postData, out string statusCode) { if (url.StartsWith("https")) ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12; var httpContent = new StringContent(postData); httpContent.Headers.ContentType = new MediaTypeHeaderValue("application/json") { CharSet = "utf-8" }; var httpClient = new HttpClient(); var response = httpClient.PostAsync(url, httpContent).Result; statusCode = response.StatusCode.ToString(); if (response.IsSuccessStatusCode) { string result = response.Content.ReadAsStringAsync().Result; return result; } return null; } /// <summary> /// POST /// </summary> /// <typeparam name="T"></typeparam> /// <param name="url"></param> /// <param name="postData"></param> /// <returns>指定对象</returns> public static T Post<T>(string url, string postData) where T : class, new() { if (url.StartsWith("https")) ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12; var httpContent = new StringContent(postData); httpContent.Headers.ContentType = new MediaTypeHeaderValue("application/json") { CharSet = "utf-8" }; var httpClient = new HttpClient(); var response = httpClient.PostAsync(url, httpContent).Result; T result = default(T); if (response.IsSuccessStatusCode) { Task<string> t = response.Content.ReadAsStringAsync(); string s = t.Result; result = JsonConvert.DeserializeObject<T>(s); } return result; } } } |
from:https://www.cnblogs.com/louby/p/8021527.html
View Details利用HttpClient进行Http请求,基于此,简单地封装了下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 |
using System; using System.Collections.Generic; using System.Collections.Specialized; using System.IO; using System.Linq; using System.Net; using System.Net.Http; using System.Text; namespace ConsoleApplication2 { public class HTTPClientHelper { private static readonly HttpClient HttpClient; static HTTPClientHelper() { var handler = new HttpClientHandler() { AutomaticDecompression = DecompressionMethods.None }; HttpClient = new HttpClient(handler); } /// <summary> /// get请求,可以对请求头进行多项设置 /// </summary> /// <param name="paramArray"></param> /// <param name="url"></param> /// <returns></returns> public static string GetResponseByGet(List<KeyValuePair<string,string>> paramArray, string url) { string result = ""; var httpclient = HTTPClientHelper.HttpClient; url = url + "?" + BuildParam(paramArray); var response = httpclient.GetAsync(url).Result; if (response.IsSuccessStatusCode) { Stream myResponseStream = response.Content.ReadAsStreamAsync().Result; StreamReader myStreamReader = new StreamReader(myResponseStream, Encoding.GetEncoding("utf-8")); result = myStreamReader.ReadToEnd(); myStreamReader.Close(); myResponseStream.Close(); } return result; } public static string GetResponseBySimpleGet(List<KeyValuePair<string,string>> paramArray, string url) { var httpclient = HTTPClientHelper.HttpClient; url = url + "?" + BuildParam(paramArray); var result = httpclient.GetStringAsync(url).Result; return result; } public static string HttpPostRequestAsync(string Url, List<KeyValuePair<string, string>> paramArray, string ContentType = "application/x-www-form-urlencoded") { string result = ""; var postData = BuildParam(paramArray); var data = Encoding.ASCII.GetBytes(postData); try { using (HttpClient http = new HttpClient()) { http.DefaultRequestHeaders.Add("User-Agent", @"Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"); http.DefaultRequestHeaders.Add("Accept", @"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"); HttpResponseMessage message = null; using (Stream dataStream = new MemoryStream(data ?? new byte[0])) { using (HttpContent content = new StreamContent(dataStream)) { content.Headers.Add("Content-Type", ContentType); var task = http.PostAsync(Url, content); message = task.Result; } } if (message != null && message.StatusCode == System.Net.HttpStatusCode.OK) { using (message) { result = message.Content.ReadAsStringAsync().Result; } } } } catch (Exception ex) { Console.WriteLine(ex.Message); } return result; } private static string Encode(string content, Encoding encode = null) { if (encode == null) return content; return System.Web.HttpUtility.UrlEncode(content, Encoding.UTF8); } private static string BuildParam(List<KeyValuePair<string, string>> paramArray, Encoding encode = null) { string url = ""; if (encode == null) encode = Encoding.UTF8; if (paramArray != null && paramArray.Count > 0) { var parms = ""; foreach (var item in paramArray) { parms += string.Format("{0}={1}&", Encode(item.Key, encode), Encode(item.Value, encode)); } if (parms != "") { parms = parms.TrimEnd('&'); } url += parms; } return url; } } } |
有关更多的Http请求,请看这里:https://github.com/wangqiang3311/HttpRequestDemo from:http://www.cnblogs.com/wangqiang3311/p/8991214.html
View Details一、 System.Net.Http.HttpClient简介 System.Net.Http 是微软.net4.5中推出的HTTP 应用程序的编程接口, 微软称之为“现代化的 HTTP 编程接口”, 主要提供如下内容: 1. 用户通过 HTTP 使用现代化的 Web Service 的客户端组件; 2. 能够同时在客户端与服务端同时使用的 HTTP 组件(比如处理 HTTP 标头和消息), 为客户端和服务端提供一致的编程模型。 个人看来是抄袭apache http client ,目前网上用的人好像不多,个人认为使用httpclient最大的好处是:不用自己管理cookie,只要负责写好请求即可。 由于网上资料不多,这里借登录博客园网站做个简单的总结其get和post请求的用法。 查看微软的api可以发现其属性方法:http://msdn.microsoft.com/zh-cn/library/system.net.http.httpclient.aspx 由其api可以看出如果想设置请求头只需要在DefaultRequestHeaders里进行设置 创建httpcliet可以直接new HttpClient() 发送请求可以按发送方式分别调用其方法,如get调用GetAsync(Uri),post调用PostAsync(Uri, HttpContent),其它依此类推。。。 二、实例(模拟post登录博客园) 首先,需要说明的是,本实例环境是win7 64位+vs 2013+ .net 4.5框架。 1.使用vs2013新建一个控制台程序,或者窗体程序,如下图所示: 2.必须引入System.Net.Http框架,否则将不能使用httpclient 3.实现代码
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 |
using System; using System.Collections.Generic; using System.IO; using System.Linq; using System.Net.Http; using System.Text; using System.Text.RegularExpressions; using System.Threading.Tasks; namespace ClassLibrary1 { public class Class1 { private static String dir = @"C:\work\"; /// <summary> /// 写文件到本地 /// </summary> /// <param name="fileName"></param> /// <param name="html"></param> public static void Write(string fileName, string html) { try { FileStream fs = new FileStream(dir + fileName, FileMode.Create); StreamWriter sw = new StreamWriter(fs, Encoding.Default); sw.Write(html); sw.Close(); fs.Close(); }catch(Exception ex){ Console.WriteLine(ex.StackTrace); } } /// <summary> /// 写文件到本地 /// </summary> /// <param name="fileName"></param> /// <param name="html"></param> public static void Write(string fileName, byte[] html) { try { File.WriteAllBytes(dir + fileName, html); } catch (Exception ex) { Console.WriteLine(ex.StackTrace); } } /// <summary> /// 登录博客园 /// </summary> public static void LoginCnblogs() { HttpClient httpClient = new HttpClient(); httpClient.MaxResponseContentBufferSize = 256000; httpClient.DefaultRequestHeaders.Add("user-agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"); String url = "http://passport.cnblogs.com/login.aspx"; HttpResponseMessage response = httpClient.GetAsync(new Uri(url)).Result; String result = response.Content.ReadAsStringAsync().Result; String username = "hi_amos"; String password = "密码"; do { String __EVENTVALIDATION = new Regex("id=\"__EVENTVALIDATION\" value=\"(.*?)\"").Match(result).Groups[1].Value; String __VIEWSTATE = new Regex("id=\"__VIEWSTATE\" value=\"(.*?)\"").Match(result).Groups[1].Value; String LBD_VCID_c_login_logincaptcha = new Regex("id=\"LBD_VCID_c_login_logincaptcha\" value=\"(.*?)\"").Match(result).Groups[1].Value; //图片验证码 url = "http://passport.cnblogs.com" + new Regex("id=\"c_login_logincaptcha_CaptchaImage\" src=\"(.*?)\"").Match(result).Groups[1].Value; response = httpClient.GetAsync(new Uri(url)).Result; Write("amosli.png", response.Content.ReadAsByteArrayAsync().Result); Console.WriteLine("输入图片验证码:"); String imgCode = "wupve";//验证码写到本地了,需要手动填写 imgCode = Console.ReadLine(); //开始登录 url = "http://passport.cnblogs.com/login.aspx"; List<KeyValuePair<String, String>> paramList = new List<KeyValuePair<String, String>>(); paramList.Add(new KeyValuePair<string, string>("__EVENTTARGET", "")); paramList.Add(new KeyValuePair<string, string>("__EVENTARGUMENT", "")); paramList.Add(new KeyValuePair<string, string>("__VIEWSTATE", __VIEWSTATE)); paramList.Add(new KeyValuePair<string, string>("__EVENTVALIDATION", __EVENTVALIDATION)); paramList.Add(new KeyValuePair<string, string>("tbUserName", username)); paramList.Add(new KeyValuePair<string, string>("tbPassword", password)); paramList.Add(new KeyValuePair<string, string>("LBD_VCID_c_login_logincaptcha", LBD_VCID_c_login_logincaptcha)); paramList.Add(new KeyValuePair<string, string>("LBD_BackWorkaround_c_login_logincaptcha", "1")); paramList.Add(new KeyValuePair<string, string>("CaptchaCodeTextBox", imgCode)); paramList.Add(new KeyValuePair<string, string>("btnLogin", "登 录")); paramList.Add(new KeyValuePair<string, string>("txtReturnUrl", "http://home.cnblogs.com/")); response = httpClient.PostAsync(new Uri(url), new FormUrlEncodedContent(paramList)).Result; result = response.Content.ReadAsStringAsync().Result; Write("myCnblogs.html",result); } while (result.Contains("验证码错误,麻烦您重新输入")); Console.WriteLine("登录成功!"); //用完要记得释放 httpClient.Dispose(); } public static void Main() { LoginCnblogs(); } } |
代码分析: 首先,从Main函数开始,调用LoginCnblogs方法; 其次,使用GET方法:
1 2 |
HttpResponseMessage response = httpClient.GetAsync(new Uri(url)).Result; String result = response.Content.ReadAsStringAsync().Result; |
再者,使用POST方法:
1 2 3 4 5 |
List<KeyValuePair<String, String>> paramList = new List<KeyValuePair<String, String>>(); paramList.Add(new KeyValuePair<string, string>("__EVENTTARGET", "")); .... response = httpClient.PostAsync(new Uri(url), new FormUrlEncodedContent(paramList)).Result; result = response.Content.ReadAsStringAsync().Result; |
最后,注意其返回值可以是string,也可以是byte[],和stream的方式,这里看你需要什么吧。 4.登录成功后的截图 1).使用浏览器登录后的截图: 2).使用Httpcliet登录后的截图: 总结,可以发现C#中HttpClient的用法和Java中非常相似,所以,说其抄袭确实不为过。 from:https://www.cnblogs.com/amosli/p/3918538.html
View Details一般有两种办法 第一种handler.UseCookies=true(默认为true),默认的会自己带上cookies,例如
1 2 3 4 5 6 7 8 9 10 11 12 |
var handler = new HttpClientHandler() { UseCookies = true }; var client = new HttpClient(handler);// { BaseAddress = baseAddress }; client.DefaultRequestHeaders.Add("user-agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0"); client.DefaultRequestHeaders.Add("Connection", "Keep-Alive"); client.DefaultRequestHeaders.Add("Keep-Alive", "timeout=600"); var content = new FormUrlEncodedContent(new[] { new KeyValuePair<string, string>("email", "xxxx"), new KeyValuePair<string, string>("password", "xxxx"), }); var result = await client.PostAsync("https://www.xxxx.com/cp/login", content); result.EnsureSuccessStatusCode(); |
这种情况post请求登陆成功后,重定向到别的页面,也会自动带上cookies。如果把handler.UseCookies设置为false,登陆后重定向的话不会自动带上cookies,则又会跳转到登陆页面。 第二种设置 handler.UseCookies = false时,则需要手动给headers上加入cookies.
1 2 3 4 5 6 |
var handler = new HttpClientHandler() { UseCookies = false}; var client = new HttpClient(handler);// { BaseAddress = baseAddress }; var message = new HttpRequestMessage(HttpMethod.Get, url); message.Headers.Add("Cookie", "session_id=7258abbd1544b6c530a9f406d3e600239bd788fb"); var result = await client.SendAsync(message); result.EnsureSuccessStatusCode(); |
如果使用场景是:抓取需要登陆后才能看到的网页数据,建议使用第一种,不需要设置任何cookies,httpclient会自动把登陆后的cookies放置到后面的请求中。 原贴 : http://www.cnblogs.com/xiaozhu39505/p/8033108.html from:https://www.cnblogs.com/refuge/p/8060142.html
View Details修改siteName为需要的重启的网站名字,将代码拷入bat文件 @echo off cd c:\Windows\System32\inetsrv appcmd stop site "siteName" appcmd start site "siteName" from:https://blog.csdn.net/a980433875/article/details/52088252
View Details