~e ddlZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z Gdde Z Gd d e ZGd d e Zd diddddd dddf dZdZddZddZddZdZedk(reej.ddyy)N) OptionParser) DeferredList) deferToThread) iteritems) networkStringc eZdZy) FetchErrorN)__name__ __module__ __qualname__5/usr/lib/python3/dist-packages/landscape/lib/fetch.pyr r srr ceZdZdZdZdZy) HTTPCodeErrorc ||_||_yN) http_codebody)selfrrs r__init__zHTTPCodeError.__init__s" rc"d|jdS)NzServer returned HTTP code drrs r__str__zHTTPCodeError.__str__s+DNN1+=>>rc$d|jddS)Nzrrs r__repr__zHTTPCodeError.__repr__s*4>>!*>rN)r r r rrrr rrrrs??rrc.eZdZdZdZdZedZy) PyCurlErrorc ||_||_yr) error_code_message)rr#messages rrzPyCurlError.__init__s$ rc<d|jdd|jS)NzError r: r#r%rs rrzPyCurlError.__str__!s *"T\\N;;rc>d|jdd|jdS)Nzr(rs rrzPyCurlError.__repr__$s"$T__Q$7s4<<.LLrc|jSr)r$rs rr%zPyCurlError.message's }}rN)r r r rrrpropertyr%r rrr!r!s& <Mrr!FiXTc ddl} t|ts|jd}t j |} t j }|| j }|j| jtt||ri|j| jd|rK|j| jt||j| j| j|r6|j!dr%|j| j"t||rF|j| j$t't)|Dcgc] \}}|d|c}}|r|j| j*d| r|j| j,d| %|j| j.t| | %|j| j0t| |j| j2d |j| j4||j| j6d|j| j8||j| j:d|j| j<|j>|j| j@d|j| jBd  |jE|jM}|jO| jP}|d k7r tS|||Scc}}w#| jF$r+}tI|jJd|jJdd}~wwxYw) aCRetrieve a URL and return the content. @param url: The url to be fetched. @param post: If true, the POST method will be used (defaults to GET). @param data: Data to be sent to the server as the POST content. @param headers: Dictionary of header => value entries to be used on the request. @param curl: A pycurl.Curl instance to use. If not provided, one will be created. @param cainfo: Path to the file with CA certificates. @param insecure: If true, perform curl using insecure option which will not attempt to verify authenticity of the peer's certificate. (Used during autodiscovery) @param follow: If True, follow HTTP redirects (default True). @param user_agent: The user-agent to set in the request. @param proxy: The proxy url to use for the request. rNzutf-8Tzhttps:r'Fs gzip,deflate)*pycurl isinstancebytesencodeioBytesIOCurlsetoptURLrstrPOST POSTFIELDSIZElen READFUNCTIONread startswithCAINFO HTTPHEADERsortedrSSL_VERIFYPEERFOLLOWLOCATION USERAGENTPROXY MAXREDIRSCONNECTTIMEOUTLOW_SPEED_LIMITLOW_SPEED_TIMENOSIGNAL WRITEFUNCTIONwriteDNS_CACHE_TIMEOUTENCODINGperformerrorr!argsgetvaluegetinfo HTTP_CODEr)urlpostdataheaderscainfocurlconnect_timeout total_timeoutinsecurefollow user_agentproxyr2outputinputkeyvalueerrs rfetchri,s> dE "{{7# ZZ F JJLE |{{} KK M#c(34  FKK&  KK,,c$i 8 KK++V[[ 9 #..* FMM=#89   39)G:L3M N> D V--.ICIt,, KG O4 <<0!&&)QVVAY//0s!L= /MM=&M88M=c(ttg|i|S)z]Retrieve a URL asynchronously. @return: A C{Deferred} resulting in the URL content. )rri)rTkwargss r fetch_asyncrls  0 0 00rc g}|D]G}t|fi|}|r|j|||r|j|||j|It |ddS)a9 Retrieve a list of URLs asynchronously. @param callback: Optionally, a function that will be fired one time for each successful URL, and will be passed its content and the URL itself. @param errback: Optionally, a function that will be fired one time for each failing URL, and will be passed the failure and the URL itself. @return: A C{DeferredList} whose callback chain will be fired as soon as all downloads have terminated. If an error occurs, the errback chain of the C{DeferredList} will be fired immediatly. T)fireOnOneErrback consumeErrors)rl addCallback addErrbackappendr)urlscallbackerrbackrkresultsrXresults rfetch_many_asyncrxsjGS+F+    x -    gs +v  $d KKrc|jdjdd}| tjj ||}|S)zReturn the last component of the given C{url}. @param url: The URL to get the filename from. @param directory: Optionally a path to prepend to the returned filename. @note: Any trailing slash in the C{url} will be removed /)rstripsplitospathjoin)rX directoryfilenames rurl_to_filenamers@zz#$$S)"-H77<< 84 Orc 8fd}fd}t|f||d|S)aJ Retrieve a list of URLs and save their content as files in a directory. @param urls: The list URLs to fetch. @param directory: The directory to save the files to, the name of the file will equal the last fragment of the URL. @param logger: Optional function to be used to log errors for failed URLs. czt|}t|d}|j||jy)N)rwb)ropenrOclose)rZrXrfdrs rrOzfetch_to_files..writes/"3)< (D !   rcbr+dj|t|j|S)Nz Couldn't fetch file from {} ({}))formatr;rg)failurerXloggers r log_errorz!fetch_to_files..log_errors2  299 &  r)rtru)rx)rsrrrkrOrs `` rfetch_to_filesrs&  D N5) Nv NNrct}|jdd|jdd|jd|j|\}\}tt ||j |j |jy) Nz--post store_true)actionz--datar,)defaultz--cainfo)rYrZr\)r add_option parse_argsprintrirYrZr\)rTparseroptionsrXs rtestrs{ ^F h|4 h+ j!''-OGVc  >>  r__main__r/)NNr)r6r~sysoptparsertwisted.internet.deferrtwisted.internet.threadsrtwisted.python.compatrr Exceptionr rr!rirlrxrrrr argvr rrrs !/2+/   ?J ? * $       ]@1L. O:   z!"r