cURL, a part of everyone's favorite UNIX tool subset, got me into a bit of trouble recently, while trying to post a relatively large file, following a common 'just curl it' - logic (so commonplace that a lot of major projects simply incorporate curlin' as a part of standard deploy procedure).
The case was posting 8Gb file on 16Gb Xen instance. While this worked quite nice on real box, on virtual box curl said hello with : out of memory
Now that seemed quite bizzare. Figurng out that the process actually gets ENOMEM, it was logical to look at curl code and figure out what's going on.
And there it was, power-of-two allocator in the file read loop : static ParameterError file2memory(char **bufp, size_t *size, FILE *file)
...

char *newbuf;
char *buffer = NULL;
size_t alloc = 512;
size_t nused = 0;
size_t nread;

do {
if(!buffer || (alloc == nused)) {

if(alloc+1 > ((size_t)-1)/2) {
if(buffer)
free(buffer);
return PARAM_NO_MEM;
}
alloc *= 2;

if((newbuf = realloc(buffer, alloc+1)) == NULL) {
if(buffer)
free(buffer);
return PARAM_NO_MEM;
}
buffer = newbuf;
}
}


Whoa :) - now apparently someone didn't expect some geniuses will try po post XX Gb files withcurl - so it's the abusers that are to blame. Stop abusing curl and do your own posts !
However, if you don't have the time to change your app, and still want to post files of the size (N,2N) Gb on a 2N Gb box, a simple hack of given form should do it :
if (alloc < ALLOC_THRESHOLD)
alloc *= 2;
alloc = alloc + ALLOC_THRESHOLD; (Where ALLOC_THRESHOLD would usually be 1Gb)
This should make allocation linear, rather than exponential, once the allocated memory passes given threshold.
Now - what does is all has to do with XEN, you might ask?.
Couple of things, actually. First off, such environment (local or any virtualized cloud platform offering xen instances) usually provide user with something like effective 2^N - penalty memory space (say 15Gb instead of 16Gb) - and that's where the impact of power of two allocator becomes apparent much sooner. Also - memory allocation policies are quite stricter and enomems are dispatched much earlier, oom killer is fast on the trigger, etc :) - so that's why the curl ooms immediately, rather than trying to make that darn realloc() after all.
Moral of the story - don't abuse standard unix tools !
Be nice to curl - do not POST binary data larger than 50% of effective RAM.
Keep it safe ! Posted by Alek at 12:23 PM 在使用lib curl 封装的HTTPClient，使用https（443端口）发请求 时 ，遇到返回 CURL E_OUT_OF_ MEMORY ，经过验证发现原因如下： 1、在初始化 curl 时 ，有且仅有一次，使用接口： CURL code ret_code = curl _global_init( CURL _GLOBAL_SSL); 2、在程序退出 时 ，有且仅有一次，使用接口： curl _global_cleanup... function request_ post ($url = '', $param = '') { if (empty($url) || empty($param)) { return false; $ post Url = $url; $ curl Post = $param; $ curl = curl _init();//初始化 curl curl _setopt($ curl , CURL OPT_URL,$ post Url);//抓取指定网页 curl _setopt($ curl , CURL OPT_HEADER, 0);//设置header curl _set curl 命令是一个利用URL规则在shell终端命令行下工作的文件 传输 工具；它支持文件的上传和下载，所以是综合 传输 工具，但按传统，习惯称 curl 为下载工具。 作为一款强力工具， curl 支持包括HTTP、HTTPS、ftp等众多协议，还支持 POST 、cookies、认证、从指定偏移处下载部分文件、用户代理字符串、限速、文件大小、进度条等特征；做网页处理流程和数据检索自动化。 语法格式： curl [参数] [网址] 常用参数： 把输出写到该文件中，保留远程文件的文件名 通过服务端配置的用户名和密码授权访问 将下载的数据写入到文件，必须使用文件的绝对地址： 2048+0 records in2048+0 records out2147483648 bytes (2.1 GB) copied, 44.2262 s, 48.6 MB/s curl : option --data-bina... 开子线程，调用 curl _easy_perform和主线程调用 curl _easy_perform，访问https连接 时 ，有几率出现错误，返回值为27，Out of memory 错误日志： * About to connect() to xxx.com.cn port 443 (#0) * Trying 54.223.100.xxx... * connected * SSL: couldn't create a context: error:140A90F1:lib(20):func(16 本文章来给大家介绍关于在window下PHP调用 curl 报内存不够( curl out of memory )错误的 解决办法 ，有碰到此类问题的朋友可参考。今天在调用新浪微博api的 时 候(官网下载的sdk)，发现发布图片微博的 时 候不成功，报错说：Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate... PHP读取微信认证超 时 ，错误如下：[error] 1548#0: *137 FastCGI sent in stderr: "PHP message: PHP Fatal error: Uncaught exception 'WxPayException' with message ' curl 出错，错误码:28' in /home/cat/html/releases/20160825025148... DESCRIPTION This man page includes most, if not all, available error codes in lib curl . Why 模拟Web页面中提交表单，用于 POST 请求 默认Content-type为application/x-www-form-urlencoded 选项的value如果是@a_file_name，表示数据来自一个文件 选项的value如果是-，表示读取stdin作为提交的数据，... curl 简介 常用示例 curl 简介 curl 是一个利用URL规则在命令行下工作的文件 传输 工具。它支持文件的上传和下载，所以是综合 传输 工具，但按传统，习惯称url为下载工具。命令参数 -a/–append 上传文件 时 ，附加到目标文件 -b/–cookie <name=string/file> cookie字符串或文件读取位置 -c/–cookie-jar <file> 操作结束后