Archives
/
readability
2
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Bug 1300697 - Reader View missed first few paragraphs on New York Times website, r=Gijs

Browse Source
pull/338/head
Evan Tseng 7 years ago committed by Gijs
parent 5b1e69bdf2
commit 15e1f03261
20 changed files with 864 additions and 831 deletions
Show Stats Download Patch File Download Diff File

21
Readability.js
Unescape Escape View File

@ -112,7 +112,7 @@ Readability.prototype = {
// All of the regular expressions in use within readability.
// Defined up here so we don't instantiate them repeatedly in loops.
REGEXPS: {
unlikelyCandidates: /banner|breadcrumbs|combx|comment|community|cover-wrap|disqus|extra|foot|header|legends|menu|modal|related|remark|replies|rss|shoutbox|sidebar|skyscraper|social|sponsor|ad-break|agegate|pagination|pager|popup|yom-remote/i,
unlikelyCandidates: /banner|breadcrumbs|combx|comment|community|cover-wrap|disqus|extra|foot|header|legends|menu|modal|related|remark|replies|rss|shoutbox|sidebar|skyscraper|social|sponsor|supplemental|ad-break|agegate|pagination|pager|popup|yom-remote/i,
okMaybeItsACandidate: /and|article|body|column|main|shadow/i,
positive: /article|body|content|entry|hentry|h-entry|main|page|pagination|post|text|blog|story/i,
negative: /hidden|^hid$| hid$| hid |^hid |banner|combx|comment|com-|contact|foot|footer|footnote|masthead|media|meta|modal|outbrain|promo|related|scroll|share|shoutbox|sidebar|skyscraper|sponsor|shopping|tags|tool|widget/i,
@ -707,6 +707,15 @@ Readability.prototype = {
}
}
// Remove empty DIV, SECTION, and HEADER nodes
if ((node.tagName === "DIV" || node.tagName === "SECTION" || node.tagName === "HEADER" ||
node.tagName === "H1" || node.tagName === "H2" || node.tagName === "H3" ||
node.tagName === "H4" || node.tagName === "H5" || node.tagName === "H6") &&
this._isEmptyElement(node)) {
node = this._removeAndGetNext(node);
continue;
}
if (this.DEFAULT_TAGS_TO_SCORE.indexOf(node.tagName) !== -1) {
elementsToScore.push(node);
}
@ -733,8 +742,6 @@ Readability.prototype = {
p.style.display = 'inline';
p.className = 'readability-styled';
node.replaceChild(p, childNode);
} else if (this._isEmptyDivElement(childNode)) {
node.replaceChild(doc.createTextNode(childNode.textContent), childNode);
}
});
}
@ -908,6 +915,9 @@ Readability.prototype = {
topCandidate = parentOfTopCandidate;
parentOfTopCandidate = topCandidate.parentNode;
}
if (!topCandidate.readability) {
this._initializeNode(topCandidate);
}
}
// Now that we have the top candidate, look through its siblings for content
@ -1160,9 +1170,8 @@ Readability.prototype = {
});
},
_isEmptyDivElement: function(node) {
_isEmptyElement: function(node) {
return node.nodeType === Node.ELEMENT_NODE &&
node.tagName === "DIV" &&
node.children.length == 0 &&
node.textContent.trim().length == 0;
},
@ -1754,7 +1763,7 @@ Readability.prototype = {
var contentLength = this._getInnerText(node).length;
var haveToRemove =
(img > 1 && img > p && !this._hasAncestorTag(node, "figure")) ||
(img > 1 && p / img < 0.5 && !this._hasAncestorTag(node, "figure")) ||
(!isList && li > p) ||
(input > Math.floor(p/3)) ||
(!isList && contentLength < 25 && (img === 0 || img > 2) && !this._hasAncestorTag(node, "figure")) ||

768
test/test-pages/002/expected.html
Unescape Escape View File

@ -1,386 +1,388 @@
<div id="readability-page-1" class="page">
<article class="post" role="article">
<p>For more than a decade the Web has used XMLHttpRequest (XHR) to achieve asynchronous requests in JavaScript. While very useful, XHR is not a very nice API. It suffers from lack of separation of concerns. The input, output and state are all managed by interacting with one object, and state is tracked using events. Also, the event-based model doesn’t play well with JavaScript’s recent focus on Promise- and generator-based asynchronous programming.</p>
<p>The <a href="https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API">Fetch API</a> intends to fix most of these problems. It does this by introducing the same primitives to JS that are used in the HTTP protocol. In addition, it introduces a utility function <code>fetch()</code> that succinctly captures the intention of retrieving a resource from the network.</p>
<p>The <a href="https://fetch.spec.whatwg.org">Fetch specification</a>, which defines the API, nails down the semantics of a user agent fetching a resource. This, combined with ServiceWorkers, is an attempt to:</p>
<ol>
<li>Improve the offline experience.</li>
<li>Expose the building blocks of the Web to the platform as part of the <a href="https://extensiblewebmanifesto.org/">extensible web movement</a>.</li>
</ol>
<p>As of this writing, the Fetch API is available in Firefox 39 (currently Nightly) and Chrome 42 (currently dev). Github has a <a href="https://github.com/github/fetch">Fetch polyfill</a>.</p>
<h2>Feature detection</h2>
<p>Fetch API support can be detected by checking for <code>Headers</code>,<code>Request</code>, <code>Response</code> or <code>fetch</code> on the <code>window</code> or <code>worker</code> scope.</p>
<h2>Simple fetching</h2>
<p>The most useful, high-level part of the Fetch API is the <code>fetch()</code> function. In its simplest form it takes a URL and returns a promise that resolves to the response. The response is captured as a <code>Response</code> object.</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript">fetch<span>(</span><span>"/data.json"</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>res<span>)</span> <span>{</span>
<span>// res instanceof Response == true.</span>
<span>if</span> <span>(</span>res.<span>ok</span><span>)</span> <span>{</span>
res.<span>json</span><span>(</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>data<span>)</span> <span>{</span>
console.<span>log</span><span>(</span>data.<span>entries</span><span>)</span><span>;</span>
<div id="content-main" class="hfeed">
<article class="post" role="article">
<p>For more than a decade the Web has used XMLHttpRequest (XHR) to achieve asynchronous requests in JavaScript. While very useful, XHR is not a very nice API. It suffers from lack of separation of concerns. The input, output and state are all managed by interacting with one object, and state is tracked using events. Also, the event-based model doesn’t play well with JavaScript’s recent focus on Promise- and generator-based asynchronous programming.</p>
<p>The <a href="https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API">Fetch API</a> intends to fix most of these problems. It does this by introducing the same primitives to JS that are used in the HTTP protocol. In addition, it introduces a utility function <code>fetch()</code> that succinctly captures the intention of retrieving a resource from the network.</p>
<p>The <a href="https://fetch.spec.whatwg.org">Fetch specification</a>, which defines the API, nails down the semantics of a user agent fetching a resource. This, combined with ServiceWorkers, is an attempt to:</p>
<ol>
<li>Improve the offline experience.</li>
<li>Expose the building blocks of the Web to the platform as part of the <a href="https://extensiblewebmanifesto.org/">extensible web movement</a>.</li>
</ol>
<p>As of this writing, the Fetch API is available in Firefox 39 (currently Nightly) and Chrome 42 (currently dev). Github has a <a href="https://github.com/github/fetch">Fetch polyfill</a>.</p>
<h2>Feature detection</h2>
<p>Fetch API support can be detected by checking for <code>Headers</code>,<code>Request</code>, <code>Response</code> or <code>fetch</code> on the <code>window</code> or <code>worker</code> scope.</p>
<h2>Simple fetching</h2>
<p>The most useful, high-level part of the Fetch API is the <code>fetch()</code> function. In its simplest form it takes a URL and returns a promise that resolves to the response. The response is captured as a <code>Response</code> object.</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript">fetch<span>(</span><span>"/data.json"</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>res<span>)</span> <span>{</span>
<span>// res instanceof Response == true.</span>
<span>if</span> <span>(</span>res.<span>ok</span><span>)</span> <span>{</span>
res.<span>json</span><span>(</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>data<span>)</span> <span>{</span>
console.<span>log</span><span>(</span>data.<span>entries</span><span>)</span><span>;</span>
<span>}</span><span>)</span><span>;</span>
<span>}</span> <span>else</span> <span>{</span>
console.<span>log</span><span>(</span><span>"Looks like the response wasn't perfect, got status"</span><span>,</span> res.<span>status</span><span>)</span><span>;</span>
<span>}</span>
<span>}</span><span>,</span> <span>function</span><span>(</span>e<span>)</span> <span>{</span>
console.<span>log</span><span>(</span><span>"Fetch failed!"</span><span>,</span> e<span>)</span><span>;</span>
<span>}</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>Submitting some parameters, it would look like this:</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript">fetch<span>(</span><span>"http://www.example.org/submit.php"</span><span>,</span> <span>{</span>
method<span>:</span> <span>"POST"</span><span>,</span>
headers<span>:</span> <span>{</span>
<span>"Content-Type"</span><span>:</span> <span>"application/x-www-form-urlencoded"</span>
<span>}</span><span>,</span>
body<span>:</span> <span>"firstName=Nikhil&amp;favColor=blue&amp;password=easytoguess"</span>
<span>}</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>res<span>)</span> <span>{</span>
<span>if</span> <span>(</span>res.<span>ok</span><span>)</span> <span>{</span>
alert<span>(</span><span>"Perfect! Your settings are saved."</span><span>)</span><span>;</span>
<span>}</span> <span>else</span> <span>if</span> <span>(</span>res.<span>status</span> <span>==</span> <span>401</span><span>)</span> <span>{</span>
alert<span>(</span><span>"Oops! You are not authorized."</span><span>)</span><span>;</span>
<span>}</span>
<span>}</span><span>,</span> <span>function</span><span>(</span>e<span>)</span> <span>{</span>
alert<span>(</span><span>"Error submitting form!"</span><span>)</span><span>;</span>
<span>}</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>The <code>fetch()</code> function’s arguments are the same as those passed to the
<br/> <code>Request()</code> constructor, so you may directly pass arbitrarily complex requests to <code>fetch()</code> as discussed below.</p>
<h2>Headers</h2>
<p>Fetch introduces 3 interfaces. These are <code>Headers</code>, <code>Request</code> and
<br/> <code>Response</code>. They map directly to the underlying HTTP concepts, but have
<br/>certain visibility filters in place for privacy and security reasons, such as
<br/>supporting CORS rules and ensuring cookies aren’t readable by third parties.</p>
<p>The <a href="https://fetch.spec.whatwg.org/#headers-class">Headers interface</a> is a simple multi-map of names to values:</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> content <span>=</span> <span>"Hello World"</span><span>;</span>
<span>var</span> reqHeaders <span>=</span> <span>new</span> Headers<span>(</span><span>)</span><span>;</span>
reqHeaders.<span>append</span><span>(</span><span>"Content-Type"</span><span>,</span> <span>"text/plain"</span>
reqHeaders.<span>append</span><span>(</span><span>"Content-Length"</span><span>,</span> content.<span>length</span>.<span>toString</span><span>(</span><span>)</span><span>)</span><span>;</span>
reqHeaders.<span>append</span><span>(</span><span>"X-Custom-Header"</span><span>,</span> <span>"ProcessThisImmediately"</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>The same can be achieved by passing an array of arrays or a JS object literal
<br/>to the constructor:</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript">reqHeaders <span>=</span> <span>new</span> Headers<span>(</span><span>{</span>
<span>"Content-Type"</span><span>:</span> <span>"text/plain"</span><span>,</span>
<span>"Content-Length"</span><span>:</span> content.<span>length</span>.<span>toString</span><span>(</span><span>)</span><span>,</span>
<span>"X-Custom-Header"</span><span>:</span> <span>"ProcessThisImmediately"</span><span>,</span>
<span>}</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>The contents can be queried and retrieved:</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript">console.<span>log</span><span>(</span>reqHeaders.<span>has</span><span>(</span><span>"Content-Type"</span><span>)</span><span>)</span><span>;</span> <span>// true</span>
console.<span>log</span><span>(</span>reqHeaders.<span>has</span><span>(</span><span>"Set-Cookie"</span><span>)</span><span>)</span><span>;</span> <span>// false</span>
reqHeaders.<span>set</span><span>(</span><span>"Content-Type"</span><span>,</span> <span>"text/html"</span><span>)</span><span>;</span>
reqHeaders.<span>append</span><span>(</span><span>"X-Custom-Header"</span><span>,</span> <span>"AnotherValue"</span><span>)</span><span>;</span>
&nbsp;
console.<span>log</span><span>(</span>reqHeaders.<span>get</span><span>(</span><span>"Content-Length"</span><span>)</span><span>)</span><span>;</span> <span>// 11</span>
console.<span>log</span><span>(</span>reqHeaders.<span>getAll</span><span>(</span><span>"X-Custom-Header"</span><span>)</span><span>)</span><span>;</span> <span>// ["ProcessThisImmediately", "AnotherValue"]</span>
&nbsp;
reqHeaders.<span>delete</span><span>(</span><span>"X-Custom-Header"</span><span>)</span><span>;</span>
console.<span>log</span><span>(</span>reqHeaders.<span>getAll</span><span>(</span><span>"X-Custom-Header"</span><span>)</span><span>)</span><span>;</span> <span>// []</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>Some of these operations are only useful in ServiceWorkers, but they provide
<br/>a much nicer API to Headers.</p>
<p>Since Headers can be sent in requests, or received in responses, and have various limitations about what information can and should be mutable, <code>Headers</code> objects have a <strong>guard</strong> property. This is not exposed to the Web, but it affects which mutation operations are allowed on the Headers object.
<br/>Possible values are:</p>
<ul>
<li>“none”: default.</li>
<li>“request”: guard for a Headers object obtained from a Request (<code>Request.headers</code>).</li>
<li>“request-no-cors”: guard for a Headers object obtained from a Request created
<br/>with mode “no-cors”.</li>
<li>“response”: naturally, for Headers obtained from Response (<code>Response.headers</code>).</li>
<li>“immutable”: Mostly used for ServiceWorkers, renders a Headers object
<br/>read-only.</li>
</ul>
<p>The details of how each guard affects the behaviors of the Headers object are
<br/>in the <a href="https://fetch.spec.whatwg.org">specification</a>. For example, you may not append or set a “request” guarded Headers’ “Content-Length” header. Similarly, inserting “Set-Cookie” into a Response header is not allowed so that ServiceWorkers may not set cookies via synthesized Responses.</p>
<p>All of the Headers methods throw TypeError if <code>name</code> is not a <a href="https://fetch.spec.whatwg.org/#concept-header-name">valid HTTP Header name</a>. The mutation operations will throw TypeError if there is an immutable guard. Otherwise they fail silently. For example:</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> res <span>=</span> Response.<span>error</span><span>(</span><span>)</span><span>;</span>
<span>try</span> <span>{</span>
res.<span>headers</span>.<span>set</span><span>(</span><span>"Origin"</span><span>,</span> <span>"http://mybank.com"</span><span>)</span><span>;</span>
<span>}</span> <span>catch</span><span>(</span>e<span>)</span> <span>{</span>
console.<span>log</span><span>(</span><span>"Cannot pretend to be a bank!"</span><span>)</span><span>;</span>
<span>}</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<h2>Request</h2>
<p>The Request interface defines a request to fetch a resource over HTTP. URL, method and headers are expected, but the Request also allows specifying a body, a request mode, credentials and cache hints.</p>
<p>The simplest Request is of course, just a URL, as you may do to GET a resource.</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> req <span>=</span> <span>new</span> Request<span>(</span><span>"/index.html"</span><span>)</span><span>;</span>
console.<span>log</span><span>(</span>req.<span>method</span><span>)</span><span>;</span> <span>// "GET"</span>
console.<span>log</span><span>(</span>req.<span>url</span><span>)</span><span>;</span> <span>// "http://example.com/index.html"</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>You may also pass a Request to the <code>Request()</code> constructor to create a copy.
<br/>(This is not the same as calling the <code>clone()</code> method, which is covered in
<br/>the “Reading bodies” section.).</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> copy <span>=</span> <span>new</span> Request<span>(</span>req<span>)</span><span>;</span>
console.<span>log</span><span>(</span>copy.<span>method</span><span>)</span><span>;</span> <span>// "GET"</span>
console.<span>log</span><span>(</span>copy.<span>url</span><span>)</span><span>;</span> <span>// "http://example.com/index.html"</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>Again, this form is probably only useful in ServiceWorkers.</p>
<p>The non-URL attributes of the <code>Request</code> can only be set by passing initial
<br/>values as a second argument to the constructor. This argument is a dictionary.</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> uploadReq <span>=</span> <span>new</span> Request<span>(</span><span>"/uploadImage"</span><span>,</span> <span>{</span>
method<span>:</span> <span>"POST"</span><span>,</span>
headers<span>:</span> <span>{</span>
<span>"Content-Type"</span><span>:</span> <span>"image/png"</span><span>,</span>
<span>}</span><span>,</span>
body<span>:</span> <span>"image data"</span>
<span>}</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>The Request’s mode is used to determine if cross-origin requests lead to valid responses, and which properties on the response are readable. Legal mode values are <code>"same-origin"</code>, <code>"no-cors"</code> (default) and <code>"cors"</code>.</p>
<p>The <code>"same-origin"</code> mode is simple, if a request is made to another origin with this mode set, the result is simply an error. You could use this to ensure that
<br/>a request is always being made to your origin.</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> arbitraryUrl <span>=</span> document.<span>getElementById</span><span>(</span><span>"url-input"</span><span>)</span>.<span>value</span><span>;</span>
fetch<span>(</span>arbitraryUrl<span>,</span> <span>{</span> mode<span>:</span> <span>"same-origin"</span> <span>}</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>res<span>)</span> <span>{</span>
console.<span>log</span><span>(</span><span>"Response succeeded?"</span><span>,</span> res.<span>ok</span><span>)</span><span>;</span>
<span>}</span><span>,</span> <span>function</span><span>(</span>e<span>)</span> <span>{</span>
console.<span>log</span><span>(</span><span>"Please enter a same-origin URL!"</span><span>)</span><span>;</span>
<span>}</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>The <code>"no-cors"</code> mode captures what the web platform does by default for scripts you import from CDNs, images hosted on other domains, and so on. First, it prevents the method from being anything other than “HEAD”, “GET” or “POST”. Second, if any ServiceWorkers intercept these requests, they may not add or override any headers except for <a href="https://fetch.spec.whatwg.org/#simple-header">these</a>. Third, JavaScript may not access any properties of the resulting Response. This ensures that ServiceWorkers do not affect the semantics of the Web and prevents security and privacy issues that could arise from leaking data across domains.</p>
<p><code>"cors"</code> mode is what you’ll usually use to make known cross-origin requests to access various APIs offered by other vendors. These are expected to adhere to
<br/>the <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS">CORS protocol</a>. Only a <a href="https://fetch.spec.whatwg.org/#concept-filtered-response-cors">limited set</a> of headers is exposed in the Response, but the body is readable. For example, you could get a list of Flickr’s <a href="https://www.flickr.com/services/api/flickr.interestingness.getList.html">most interesting</a> photos today like this:</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> u <span>=</span> <span>new</span> URLSearchParams<span>(</span><span>)</span><span>;</span>
u.<span>append</span><span>(</span><span>'method'</span><span>,</span> <span>'flickr.interestingness.getList'</span><span>)</span><span>;</span>
u.<span>append</span><span>(</span><span>'api_key'</span><span>,</span> <span>'&lt;insert api key here&gt;'</span><span>)</span><span>;</span>
u.<span>append</span><span>(</span><span>'format'</span><span>,</span> <span>'json'</span><span>)</span><span>;</span>
u.<span>append</span><span>(</span><span>'nojsoncallback'</span><span>,</span> <span>'1'</span><span>)</span><span>;</span>
&nbsp;
<span>var</span> apiCall <span>=</span> fetch<span>(</span><span>'https://api.flickr.com/services/rest?'</span> <span>+</span> u<span>)</span><span>;</span>
&nbsp;
apiCall.<span>then</span><span>(</span><span>function</span><span>(</span>response<span>)</span> <span>{</span>
<span>return</span> response.<span>json</span><span>(</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>json<span>)</span> <span>{</span>
<span>// photo is a list of photos.</span>
<span>return</span> json.<span>photos</span>.<span>photo</span><span>;</span>
<span>}</span><span>)</span><span>;</span>
<span>}</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>photos<span>)</span> <span>{</span>
photos.<span>forEach</span><span>(</span><span>function</span><span>(</span>photo<span>)</span> <span>{</span>
console.<span>log</span><span>(</span>photo.<span>title</span><span>)</span><span>;</span>
<span>}</span><span>)</span><span>;</span>
<span>}</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>You may not read out the “Date” header since Flickr does not allow it via
<br/> <code>Access-Control-Expose-Headers</code>.</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript">response.<span>headers</span>.<span>get</span><span>(</span><span>"Date"</span><span>)</span><span>;</span> <span>// null</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>The <code>credentials</code> enumeration determines if cookies for the other domain are
<br/>sent to cross-origin requests. This is similar to XHR’s <code>withCredentials</code>
<br/>flag, but tri-valued as <code>"omit"</code> (default), <code>"same-origin"</code> and <code>"include"</code>.</p>
<p>The Request object will also give the ability to offer caching hints to the user-agent. This is currently undergoing some <a href="https://github.com/slightlyoff/ServiceWorker/issues/585">security review</a>. Firefox exposes the attribute, but it has no effect.</p>
<p>Requests have two read-only attributes that are relevant to ServiceWorkers
<br/>intercepting them. There is the string <code>referrer</code>, which is set by the UA to be
<br/>the referrer of the Request. This may be an empty string. The other is
<br/> <code>context</code> which is a rather <a href="https://fetch.spec.whatwg.org/#requestcredentials">large enumeration</a> defining what sort of resource is being fetched. This could be “image” if the request is from an &lt;img&gt;tag in the controlled document, “worker” if it is an attempt to load a worker script, and so on. When used with the <code>fetch()</code> function, it is “fetch”.</p>
<h2>Response</h2>
<p><code>Response</code> instances are returned by calls to <code>fetch()</code>. They can also be created by JS, but this is only useful in ServiceWorkers.</p>
<p>We have already seen some attributes of Response when we looked at <code>fetch()</code>. The most obvious candidates are <code>status</code>, an integer (default value 200) and <code>statusText</code> (default value “OK”), which correspond to the HTTP status code and reason. The <code>ok</code> attribute is just a shorthand for checking that <code>status</code> is in the range 200-299 inclusive.</p>
<p><code>headers</code> is the Response’s Headers object, with guard “response”. The <code>url</code> attribute reflects the URL of the corresponding request.</p>
<p>Response also has a <code>type</code>, which is “basic”, “cors”, “default”, “error” or
<br/>“opaque”.</p>
<ul>
<li><code>"basic"</code>: normal, same origin response, with all headers exposed except
<br/>“Set-Cookie” and “Set-Cookie2″.</li>
<li><code>"cors"</code>: response was received from a valid cross-origin request. <a href="https://fetch.spec.whatwg.org/#concept-filtered-response-cors">Certain headers and the body</a>may be accessed.</li>
<li><code>"error"</code>: network error. No useful information describing the error is available. The Response’s status is 0, headers are empty and immutable. This is the type for a Response obtained from <code>Response.error()</code>.</li>
<li><code>"opaque"</code>: response for “no-cors” request to cross-origin resource. <a href="https://fetch.spec.whatwg.org/#concept-filtered-response-opaque">Severely<br/>
restricted</a> </li>
</ul>
<p>The “error” type results in the <code>fetch()</code> Promise rejecting with TypeError.</p>
<p>There are certain attributes that are useful only in a ServiceWorker scope. The
<br/>idiomatic way to return a Response to an intercepted request in ServiceWorkers is:</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript">addEventListener<span>(</span><span>'fetch'</span><span>,</span> <span>function</span><span>(</span>event<span>)</span> <span>{</span>
event.<span>respondWith</span><span>(</span><span>new</span> Response<span>(</span><span>"Response body"</span><span>,</span> <span>{</span>
headers<span>:</span> <span>{</span> <span>"Content-Type"</span> <span>:</span> <span>"text/plain"</span> <span>}</span>
<span>}</span><span>)</span><span>;</span>
<span>}</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>As you can see, Response has a two argument constructor, where both arguments are optional. The first argument is a body initializer, and the second is a dictionary to set the <code>status</code>, <code>statusText</code> and <code>headers</code>.</p>
<p>The static method <code>Response.error()</code> simply returns an error response. Similarly, <code>Response.redirect(url, status)</code> returns a Response resulting in
<br/>a redirect to <code>url</code>.</p>
<h2>Dealing with bodies</h2>
<p>Both Requests and Responses may contain body data. We’ve been glossing over it because of the various data types body may contain, but we will cover it in detail now.</p>
<p>A body is an instance of any of the following types.</p>
<ul>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer">ArrayBuffer</a> </li>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/ArrayBufferView">ArrayBufferView</a> (Uint8Array and friends)</li>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Blob">Blob</a>/ <a href="https://developer.mozilla.org/en-US/docs/Web/API/File">File</a> </li>
<li>string</li>
<li><a href="https://url.spec.whatwg.org/#interface-urlsearchparams">URLSearchParams</a> </li>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/FormData">FormData</a> – currently not supported by either Gecko or Blink. Firefox expects to ship this in version 39 along with the rest of Fetch.</li>
</ul>
<p>In addition, Request and Response both offer the following methods to extract their body. These all return a Promise that is eventually resolved with the actual content.</p>
<ul>
<li><code>arrayBuffer()</code> </li>
<li><code>blob()</code> </li>
<li><code>json()</code> </li>
<li><code>text()</code> </li>
<li><code>formData()</code> </li>
</ul>
<p>This is a significant improvement over XHR in terms of ease of use of non-text data!</p>
<p>Request bodies can be set by passing <code>body</code> parameters:</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> form <span>=</span> <span>new</span> FormData<span>(</span>document.<span>getElementById</span><span>(</span><span>'login-form'</span><span>)</span><span>)</span><span>;</span>
fetch<span>(</span><span>"/login"</span><span>,</span> <span>{</span>
method<span>:</span> <span>"POST"</span><span>,</span>
body<span>:</span> form
<span>}</span><span>)</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>Responses take the first argument as the body.</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> res <span>=</span> <span>new</span> Response<span>(</span><span>new</span> File<span>(</span><span>[</span><span>"chunk"</span><span>,</span> <span>"chunk"</span><span>]</span><span>,</span> <span>"archive.zip"</span><span>,</span>
<span>{</span> type<span>:</span> <span>"application/zip"</span> <span>}</span><span>)</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>Both Request and Response (and by extension the <code>fetch()</code> function), will try to intelligently <a href="https://fetch.spec.whatwg.org/#concept-bodyinit-extract">determine the content type</a>. Request will also automatically set a “Content-Type” header if none is set in the dictionary.</p>
<h3>Streams and cloning</h3>
<p>It is important to realise that Request and Response bodies can only be read once! Both interfaces have a boolean attribute <code>bodyUsed</code> to determine if it is safe to read or not.</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> res <span>=</span> <span>new</span> Response<span>(</span><span>"one time use"</span><span>)</span><span>;</span>
console.<span>log</span><span>(</span>res.<span>bodyUsed</span><span>)</span><span>;</span> <span>// false</span>
res.<span>text</span><span>(</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>v<span>)</span> <span>{</span>
console.<span>log</span><span>(</span>res.<span>bodyUsed</span><span>)</span><span>;</span> <span>// true</span>
<span>}</span><span>)</span><span>;</span>
<span>}</span> <span>else</span> <span>{</span>
console.<span>log</span><span>(</span><span>"Looks like the response wasn't perfect, got status"</span><span>,</span> res.<span>status</span><span>)</span><span>;</span>
<span>}</span>
<span>}</span><span>,</span> <span>function</span><span>(</span>e<span>)</span> <span>{</span>
console.<span>log</span><span>(</span><span>"Fetch failed!"</span><span>,</span> e<span>)</span><span>;</span>
<span>}</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>Submitting some parameters, it would look like this:</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript">fetch<span>(</span><span>"http://www.example.org/submit.php"</span><span>,</span> <span>{</span>
method<span>:</span> <span>"POST"</span><span>,</span>
headers<span>:</span> <span>{</span>
<span>"Content-Type"</span><span>:</span> <span>"application/x-www-form-urlencoded"</span>
<span>}</span><span>,</span>
body<span>:</span> <span>"firstName=Nikhil&amp;favColor=blue&amp;password=easytoguess"</span>
<span>}</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>res<span>)</span> <span>{</span>
<span>if</span> <span>(</span>res.<span>ok</span><span>)</span> <span>{</span>
alert<span>(</span><span>"Perfect! Your settings are saved."</span><span>)</span><span>;</span>
<span>}</span> <span>else</span> <span>if</span> <span>(</span>res.<span>status</span> <span>==</span> <span>401</span><span>)</span> <span>{</span>
alert<span>(</span><span>"Oops! You are not authorized."</span><span>)</span><span>;</span>
<span>}</span>
<span>}</span><span>,</span> <span>function</span><span>(</span>e<span>)</span> <span>{</span>
alert<span>(</span><span>"Error submitting form!"</span><span>)</span><span>;</span>
<span>}</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>The <code>fetch()</code> function’s arguments are the same as those passed to the
<br/> <code>Request()</code> constructor, so you may directly pass arbitrarily complex requests to <code>fetch()</code> as discussed below.</p>
<h2>Headers</h2>
<p>Fetch introduces 3 interfaces. These are <code>Headers</code>, <code>Request</code> and
<br/> <code>Response</code>. They map directly to the underlying HTTP concepts, but have
<br/>certain visibility filters in place for privacy and security reasons, such as
<br/>supporting CORS rules and ensuring cookies aren’t readable by third parties.</p>
<p>The <a href="https://fetch.spec.whatwg.org/#headers-class">Headers interface</a> is a simple multi-map of names to values:</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> content <span>=</span> <span>"Hello World"</span><span>;</span>
<span>var</span> reqHeaders <span>=</span> <span>new</span> Headers<span>(</span><span>)</span><span>;</span>
reqHeaders.<span>append</span><span>(</span><span>"Content-Type"</span><span>,</span> <span>"text/plain"</span>
reqHeaders.<span>append</span><span>(</span><span>"Content-Length"</span><span>,</span> content.<span>length</span>.<span>toString</span><span>(</span><span>)</span><span>)</span><span>;</span>
reqHeaders.<span>append</span><span>(</span><span>"X-Custom-Header"</span><span>,</span> <span>"ProcessThisImmediately"</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>The same can be achieved by passing an array of arrays or a JS object literal
<br/>to the constructor:</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript">reqHeaders <span>=</span> <span>new</span> Headers<span>(</span><span>{</span>
<span>"Content-Type"</span><span>:</span> <span>"text/plain"</span><span>,</span>
<span>"Content-Length"</span><span>:</span> content.<span>length</span>.<span>toString</span><span>(</span><span>)</span><span>,</span>
<span>"X-Custom-Header"</span><span>:</span> <span>"ProcessThisImmediately"</span><span>,</span>
<span>}</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>The contents can be queried and retrieved:</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript">console.<span>log</span><span>(</span>reqHeaders.<span>has</span><span>(</span><span>"Content-Type"</span><span>)</span><span>)</span><span>;</span> <span>// true</span>
console.<span>log</span><span>(</span>reqHeaders.<span>has</span><span>(</span><span>"Set-Cookie"</span><span>)</span><span>)</span><span>;</span> <span>// false</span>
reqHeaders.<span>set</span><span>(</span><span>"Content-Type"</span><span>,</span> <span>"text/html"</span><span>)</span><span>;</span>
reqHeaders.<span>append</span><span>(</span><span>"X-Custom-Header"</span><span>,</span> <span>"AnotherValue"</span><span>)</span><span>;</span>
&nbsp;
console.<span>log</span><span>(</span>reqHeaders.<span>get</span><span>(</span><span>"Content-Length"</span><span>)</span><span>)</span><span>;</span> <span>// 11</span>
console.<span>log</span><span>(</span>reqHeaders.<span>getAll</span><span>(</span><span>"X-Custom-Header"</span><span>)</span><span>)</span><span>;</span> <span>// ["ProcessThisImmediately", "AnotherValue"]</span>
&nbsp;
reqHeaders.<span>delete</span><span>(</span><span>"X-Custom-Header"</span><span>)</span><span>;</span>
console.<span>log</span><span>(</span>reqHeaders.<span>getAll</span><span>(</span><span>"X-Custom-Header"</span><span>)</span><span>)</span><span>;</span> <span>// []</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>Some of these operations are only useful in ServiceWorkers, but they provide
<br/>a much nicer API to Headers.</p>
<p>Since Headers can be sent in requests, or received in responses, and have various limitations about what information can and should be mutable, <code>Headers</code> objects have a <strong>guard</strong> property. This is not exposed to the Web, but it affects which mutation operations are allowed on the Headers object.
<br/>Possible values are:</p>
<ul>
<li>“none”: default.</li>
<li>“request”: guard for a Headers object obtained from a Request (<code>Request.headers</code>).</li>
<li>“request-no-cors”: guard for a Headers object obtained from a Request created
<br/>with mode “no-cors”.</li>
<li>“response”: naturally, for Headers obtained from Response (<code>Response.headers</code>).</li>
<li>“immutable”: Mostly used for ServiceWorkers, renders a Headers object
<br/>read-only.</li>
</ul>
<p>The details of how each guard affects the behaviors of the Headers object are
<br/>in the <a href="https://fetch.spec.whatwg.org">specification</a>. For example, you may not append or set a “request” guarded Headers’ “Content-Length” header. Similarly, inserting “Set-Cookie” into a Response header is not allowed so that ServiceWorkers may not set cookies via synthesized Responses.</p>
<p>All of the Headers methods throw TypeError if <code>name</code> is not a <a href="https://fetch.spec.whatwg.org/#concept-header-name">valid HTTP Header name</a>. The mutation operations will throw TypeError if there is an immutable guard. Otherwise they fail silently. For example:</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> res <span>=</span> Response.<span>error</span><span>(</span><span>)</span><span>;</span>
<span>try</span> <span>{</span>
res.<span>headers</span>.<span>set</span><span>(</span><span>"Origin"</span><span>,</span> <span>"http://mybank.com"</span><span>)</span><span>;</span>
<span>}</span> <span>catch</span><span>(</span>e<span>)</span> <span>{</span>
console.<span>log</span><span>(</span><span>"Cannot pretend to be a bank!"</span><span>)</span><span>;</span>
<span>}</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<h2>Request</h2>
<p>The Request interface defines a request to fetch a resource over HTTP. URL, method and headers are expected, but the Request also allows specifying a body, a request mode, credentials and cache hints.</p>
<p>The simplest Request is of course, just a URL, as you may do to GET a resource.</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> req <span>=</span> <span>new</span> Request<span>(</span><span>"/index.html"</span><span>)</span><span>;</span>
console.<span>log</span><span>(</span>req.<span>method</span><span>)</span><span>;</span> <span>// "GET"</span>
console.<span>log</span><span>(</span>req.<span>url</span><span>)</span><span>;</span> <span>// "http://example.com/index.html"</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>You may also pass a Request to the <code>Request()</code> constructor to create a copy.
<br/>(This is not the same as calling the <code>clone()</code> method, which is covered in
<br/>the “Reading bodies” section.).</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> copy <span>=</span> <span>new</span> Request<span>(</span>req<span>)</span><span>;</span>
console.<span>log</span><span>(</span>copy.<span>method</span><span>)</span><span>;</span> <span>// "GET"</span>
console.<span>log</span><span>(</span>copy.<span>url</span><span>)</span><span>;</span> <span>// "http://example.com/index.html"</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>Again, this form is probably only useful in ServiceWorkers.</p>
<p>The non-URL attributes of the <code>Request</code> can only be set by passing initial
<br/>values as a second argument to the constructor. This argument is a dictionary.</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> uploadReq <span>=</span> <span>new</span> Request<span>(</span><span>"/uploadImage"</span><span>,</span> <span>{</span>
method<span>:</span> <span>"POST"</span><span>,</span>
headers<span>:</span> <span>{</span>
<span>"Content-Type"</span><span>:</span> <span>"image/png"</span><span>,</span>
<span>}</span><span>,</span>
body<span>:</span> <span>"image data"</span>
<span>}</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>The Request’s mode is used to determine if cross-origin requests lead to valid responses, and which properties on the response are readable. Legal mode values are <code>"same-origin"</code>, <code>"no-cors"</code> (default) and <code>"cors"</code>.</p>
<p>The <code>"same-origin"</code> mode is simple, if a request is made to another origin with this mode set, the result is simply an error. You could use this to ensure that
<br/>a request is always being made to your origin.</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> arbitraryUrl <span>=</span> document.<span>getElementById</span><span>(</span><span>"url-input"</span><span>)</span>.<span>value</span><span>;</span>
fetch<span>(</span>arbitraryUrl<span>,</span> <span>{</span> mode<span>:</span> <span>"same-origin"</span> <span>}</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>res<span>)</span> <span>{</span>
console.<span>log</span><span>(</span><span>"Response succeeded?"</span><span>,</span> res.<span>ok</span><span>)</span><span>;</span>
<span>}</span><span>,</span> <span>function</span><span>(</span>e<span>)</span> <span>{</span>
console.<span>log</span><span>(</span><span>"Please enter a same-origin URL!"</span><span>)</span><span>;</span>
<span>}</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>The <code>"no-cors"</code> mode captures what the web platform does by default for scripts you import from CDNs, images hosted on other domains, and so on. First, it prevents the method from being anything other than “HEAD”, “GET” or “POST”. Second, if any ServiceWorkers intercept these requests, they may not add or override any headers except for <a href="https://fetch.spec.whatwg.org/#simple-header">these</a>. Third, JavaScript may not access any properties of the resulting Response. This ensures that ServiceWorkers do not affect the semantics of the Web and prevents security and privacy issues that could arise from leaking data across domains.</p>
<p><code>"cors"</code> mode is what you’ll usually use to make known cross-origin requests to access various APIs offered by other vendors. These are expected to adhere to
<br/>the <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS">CORS protocol</a>. Only a <a href="https://fetch.spec.whatwg.org/#concept-filtered-response-cors">limited set</a> of headers is exposed in the Response, but the body is readable. For example, you could get a list of Flickr’s <a href="https://www.flickr.com/services/api/flickr.interestingness.getList.html">most interesting</a> photos today like this:</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> u <span>=</span> <span>new</span> URLSearchParams<span>(</span><span>)</span><span>;</span>
u.<span>append</span><span>(</span><span>'method'</span><span>,</span> <span>'flickr.interestingness.getList'</span><span>)</span><span>;</span>
u.<span>append</span><span>(</span><span>'api_key'</span><span>,</span> <span>'&lt;insert api key here&gt;'</span><span>)</span><span>;</span>
u.<span>append</span><span>(</span><span>'format'</span><span>,</span> <span>'json'</span><span>)</span><span>;</span>
u.<span>append</span><span>(</span><span>'nojsoncallback'</span><span>,</span> <span>'1'</span><span>)</span><span>;</span>
&nbsp;
<span>var</span> apiCall <span>=</span> fetch<span>(</span><span>'https://api.flickr.com/services/rest?'</span> <span>+</span> u<span>)</span><span>;</span>
&nbsp;
apiCall.<span>then</span><span>(</span><span>function</span><span>(</span>response<span>)</span> <span>{</span>
<span>return</span> response.<span>json</span><span>(</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>json<span>)</span> <span>{</span>
<span>// photo is a list of photos.</span>
<span>return</span> json.<span>photos</span>.<span>photo</span><span>;</span>
<span>}</span><span>)</span><span>;</span>
<span>}</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>photos<span>)</span> <span>{</span>
photos.<span>forEach</span><span>(</span><span>function</span><span>(</span>photo<span>)</span> <span>{</span>
console.<span>log</span><span>(</span>photo.<span>title</span><span>)</span><span>;</span>
<span>}</span><span>)</span><span>;</span>
<span>}</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>You may not read out the “Date” header since Flickr does not allow it via
<br/> <code>Access-Control-Expose-Headers</code>.</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript">response.<span>headers</span>.<span>get</span><span>(</span><span>"Date"</span><span>)</span><span>;</span> <span>// null</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>The <code>credentials</code> enumeration determines if cookies for the other domain are
<br/>sent to cross-origin requests. This is similar to XHR’s <code>withCredentials</code>
<br/>flag, but tri-valued as <code>"omit"</code> (default), <code>"same-origin"</code> and <code>"include"</code>.</p>
<p>The Request object will also give the ability to offer caching hints to the user-agent. This is currently undergoing some <a href="https://github.com/slightlyoff/ServiceWorker/issues/585">security review</a>. Firefox exposes the attribute, but it has no effect.</p>
<p>Requests have two read-only attributes that are relevant to ServiceWorkers
<br/>intercepting them. There is the string <code>referrer</code>, which is set by the UA to be
<br/>the referrer of the Request. This may be an empty string. The other is
<br/> <code>context</code> which is a rather <a href="https://fetch.spec.whatwg.org/#requestcredentials">large enumeration</a> defining what sort of resource is being fetched. This could be “image” if the request is from an &lt;img&gt;tag in the controlled document, “worker” if it is an attempt to load a worker script, and so on. When used with the <code>fetch()</code> function, it is “fetch”.</p>
<h2>Response</h2>
<p><code>Response</code> instances are returned by calls to <code>fetch()</code>. They can also be created by JS, but this is only useful in ServiceWorkers.</p>
<p>We have already seen some attributes of Response when we looked at <code>fetch()</code>. The most obvious candidates are <code>status</code>, an integer (default value 200) and <code>statusText</code> (default value “OK”), which correspond to the HTTP status code and reason. The <code>ok</code> attribute is just a shorthand for checking that <code>status</code> is in the range 200-299 inclusive.</p>
<p><code>headers</code> is the Response’s Headers object, with guard “response”. The <code>url</code> attribute reflects the URL of the corresponding request.</p>
<p>Response also has a <code>type</code>, which is “basic”, “cors”, “default”, “error” or
<br/>“opaque”.</p>
<ul>
<li><code>"basic"</code>: normal, same origin response, with all headers exposed except
<br/>“Set-Cookie” and “Set-Cookie2″.</li>
<li><code>"cors"</code>: response was received from a valid cross-origin request. <a href="https://fetch.spec.whatwg.org/#concept-filtered-response-cors">Certain headers and the body</a>may be accessed.</li>
<li><code>"error"</code>: network error. No useful information describing the error is available. The Response’s status is 0, headers are empty and immutable. This is the type for a Response obtained from <code>Response.error()</code>.</li>
<li><code>"opaque"</code>: response for “no-cors” request to cross-origin resource. <a href="https://fetch.spec.whatwg.org/#concept-filtered-response-opaque">Severely<br/>
restricted</a> </li>
</ul>
<p>The “error” type results in the <code>fetch()</code> Promise rejecting with TypeError.</p>
<p>There are certain attributes that are useful only in a ServiceWorker scope. The
<br/>idiomatic way to return a Response to an intercepted request in ServiceWorkers is:</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript">addEventListener<span>(</span><span>'fetch'</span><span>,</span> <span>function</span><span>(</span>event<span>)</span> <span>{</span>
event.<span>respondWith</span><span>(</span><span>new</span> Response<span>(</span><span>"Response body"</span><span>,</span> <span>{</span>
headers<span>:</span> <span>{</span> <span>"Content-Type"</span> <span>:</span> <span>"text/plain"</span> <span>}</span>
<span>}</span><span>)</span><span>;</span>
<span>}</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>As you can see, Response has a two argument constructor, where both arguments are optional. The first argument is a body initializer, and the second is a dictionary to set the <code>status</code>, <code>statusText</code> and <code>headers</code>.</p>
<p>The static method <code>Response.error()</code> simply returns an error response. Similarly, <code>Response.redirect(url, status)</code> returns a Response resulting in
<br/>a redirect to <code>url</code>.</p>
<h2>Dealing with bodies</h2>
<p>Both Requests and Responses may contain body data. We’ve been glossing over it because of the various data types body may contain, but we will cover it in detail now.</p>
<p>A body is an instance of any of the following types.</p>
<ul>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer">ArrayBuffer</a> </li>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/ArrayBufferView">ArrayBufferView</a> (Uint8Array and friends)</li>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Blob">Blob</a>/ <a href="https://developer.mozilla.org/en-US/docs/Web/API/File">File</a> </li>
<li>string</li>
<li><a href="https://url.spec.whatwg.org/#interface-urlsearchparams">URLSearchParams</a> </li>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/FormData">FormData</a> – currently not supported by either Gecko or Blink. Firefox expects to ship this in version 39 along with the rest of Fetch.</li>
</ul>
<p>In addition, Request and Response both offer the following methods to extract their body. These all return a Promise that is eventually resolved with the actual content.</p>
<ul>
<li><code>arrayBuffer()</code> </li>
<li><code>blob()</code> </li>
<li><code>json()</code> </li>
<li><code>text()</code> </li>
<li><code>formData()</code> </li>
</ul>
<p>This is a significant improvement over XHR in terms of ease of use of non-text data!</p>
<p>Request bodies can be set by passing <code>body</code> parameters:</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> form <span>=</span> <span>new</span> FormData<span>(</span>document.<span>getElementById</span><span>(</span><span>'login-form'</span><span>)</span><span>)</span><span>;</span>
fetch<span>(</span><span>"/login"</span><span>,</span> <span>{</span>
method<span>:</span> <span>"POST"</span><span>,</span>
body<span>:</span> form
<span>}</span><span>)</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>Responses take the first argument as the body.</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> res <span>=</span> <span>new</span> Response<span>(</span><span>new</span> File<span>(</span><span>[</span><span>"chunk"</span><span>,</span> <span>"chunk"</span><span>]</span><span>,</span> <span>"archive.zip"</span><span>,</span>
<span>{</span> type<span>:</span> <span>"application/zip"</span> <span>}</span><span>)</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>Both Request and Response (and by extension the <code>fetch()</code> function), will try to intelligently <a href="https://fetch.spec.whatwg.org/#concept-bodyinit-extract">determine the content type</a>. Request will also automatically set a “Content-Type” header if none is set in the dictionary.</p>
<h3>Streams and cloning</h3>
<p>It is important to realise that Request and Response bodies can only be read once! Both interfaces have a boolean attribute <code>bodyUsed</code> to determine if it is safe to read or not.</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript"><span>var</span> res <span>=</span> <span>new</span> Response<span>(</span><span>"one time use"</span><span>)</span><span>;</span>
console.<span>log</span><span>(</span>res.<span>bodyUsed</span><span>)</span><span>;</span> <span>// false</span>
res.<span>text</span><span>(</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>v<span>)</span> <span>{</span>
console.<span>log</span><span>(</span>res.<span>bodyUsed</span><span>)</span><span>;</span> <span>// true</span>
<span>}</span><span>)</span><span>;</span>
console.<span>log</span><span>(</span>res.<span>bodyUsed</span><span>)</span><span>;</span> <span>// true</span>
&nbsp;
res.<span>text</span><span>(</span><span>)</span>.<span>catch</span><span>(</span><span>function</span><span>(</span>e<span>)</span> <span>{</span>
console.<span>log</span><span>(</span><span>"Tried to read already consumed Response"</span><span>)</span><span>;</span>
<span>}</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>This decision allows easing the transition to an eventual <a href="https://streams.spec.whatwg.org/">stream-based</a> Fetch API. The intention is to let applications consume data as it arrives, allowing for JavaScript to deal with larger files like videos, and perform things like compression and editing on the fly.</p>
<p>Often, you’ll want access to the body multiple times. For example, you can use the upcoming <a href="http://slightlyoff.github.io/ServiceWorker/spec/service_worker/index.html#cache-objects">Cache API</a> to store Requests and Responses for offline use, and Cache requires bodies to be available for reading.</p>
<p>So how do you read out the body multiple times within such constraints? The API provides a <code>clone()</code> method on the two interfaces. This will return a clone of the object, with a ‘new’ body. <code>clone()</code> MUST be called before the body of the corresponding object has been used. That is, <code>clone()</code> first, read later.</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript">addEventListener<span>(</span><span>'fetch'</span><span>,</span> <span>function</span><span>(</span>evt<span>)</span> <span>{</span>
<span>var</span> sheep <span>=</span> <span>new</span> Response<span>(</span><span>"Dolly"</span><span>)</span><span>;</span>
console.<span>log</span><span>(</span>sheep.<span>bodyUsed</span><span>)</span><span>;</span> <span>// false</span>
<span>var</span> clone <span>=</span> sheep.<span>clone</span><span>(</span><span>)</span><span>;</span>
console.<span>log</span><span>(</span>clone.<span>bodyUsed</span><span>)</span><span>;</span> <span>// false</span>
&nbsp;
clone.<span>text</span><span>(</span><span>)</span><span>;</span>
console.<span>log</span><span>(</span>sheep.<span>bodyUsed</span><span>)</span><span>;</span> <span>// false</span>
console.<span>log</span><span>(</span>clone.<span>bodyUsed</span><span>)</span><span>;</span> <span>// true</span>
&nbsp;
evt.<span>respondWith</span><span>(</span>cache.<span>add</span><span>(</span>sheep.<span>clone</span><span>(</span><span>)</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>e<span>)</span> <span>{</span>
<span>return</span> sheep<span>;</span>
<span>}</span><span>)</span><span>;</span>
<span>}</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<h2>Future improvements</h2>
<p>Along with the transition to streams, Fetch will eventually have the ability to abort running <code>fetch()</code>es and some way to report the progress of a fetch. These are provided by XHR, but are a little tricky to fit in the Promise-based nature of the Fetch API.</p>
<p>You can contribute to the evolution of this API by participating in discussions on the <a href="https://whatwg.org/mailing-list">WHATWG mailing list</a> and in the issues in the <a href="https://www.w3.org/Bugs/Public/buglist.cgi?product=WHATWG&amp;component=Fetch&amp;resolution=---">Fetch</a> and <a href="https://github.com/slightlyoff/ServiceWorker/issues">ServiceWorker</a>specifications.</p>
<p>For a better web!</p>
<p><em>The author would like to thank Andrea Marchesini, Anne van Kesteren and Ben<br/>
Kelly for helping with the specification and implementation.</em> </p>
</article>
console.<span>log</span><span>(</span>res.<span>bodyUsed</span><span>)</span><span>;</span> <span>// true</span>
&nbsp;
res.<span>text</span><span>(</span><span>)</span>.<span>catch</span><span>(</span><span>function</span><span>(</span>e<span>)</span> <span>{</span>
console.<span>log</span><span>(</span><span>"Tried to read already consumed Response"</span><span>)</span><span>;</span>
<span>}</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<p>This decision allows easing the transition to an eventual <a href="https://streams.spec.whatwg.org/">stream-based</a> Fetch API. The intention is to let applications consume data as it arrives, allowing for JavaScript to deal with larger files like videos, and perform things like compression and editing on the fly.</p>
<p>Often, you’ll want access to the body multiple times. For example, you can use the upcoming <a href="http://slightlyoff.github.io/ServiceWorker/spec/service_worker/index.html#cache-objects">Cache API</a> to store Requests and Responses for offline use, and Cache requires bodies to be available for reading.</p>
<p>So how do you read out the body multiple times within such constraints? The API provides a <code>clone()</code> method on the two interfaces. This will return a clone of the object, with a ‘new’ body. <code>clone()</code> MUST be called before the body of the corresponding object has been used. That is, <code>clone()</code> first, read later.</p>
<div class="wp_syntax">
<table>
<tbody>
<tr>
<td class="code"><pre class="javascript">addEventListener<span>(</span><span>'fetch'</span><span>,</span> <span>function</span><span>(</span>evt<span>)</span> <span>{</span>
<span>var</span> sheep <span>=</span> <span>new</span> Response<span>(</span><span>"Dolly"</span><span>)</span><span>;</span>
console.<span>log</span><span>(</span>sheep.<span>bodyUsed</span><span>)</span><span>;</span> <span>// false</span>
<span>var</span> clone <span>=</span> sheep.<span>clone</span><span>(</span><span>)</span><span>;</span>
console.<span>log</span><span>(</span>clone.<span>bodyUsed</span><span>)</span><span>;</span> <span>// false</span>
&nbsp;
clone.<span>text</span><span>(</span><span>)</span><span>;</span>
console.<span>log</span><span>(</span>sheep.<span>bodyUsed</span><span>)</span><span>;</span> <span>// false</span>
console.<span>log</span><span>(</span>clone.<span>bodyUsed</span><span>)</span><span>;</span> <span>// true</span>
&nbsp;
evt.<span>respondWith</span><span>(</span>cache.<span>add</span><span>(</span>sheep.<span>clone</span><span>(</span><span>)</span><span>)</span>.<span>then</span><span>(</span><span>function</span><span>(</span>e<span>)</span> <span>{</span>
<span>return</span> sheep<span>;</span>
<span>}</span><span>)</span><span>;</span>
<span>}</span><span>)</span><span>;</span></pre> </td>
</tr>
</tbody>
</table>
</div>
<h2>Future improvements</h2>
<p>Along with the transition to streams, Fetch will eventually have the ability to abort running <code>fetch()</code>es and some way to report the progress of a fetch. These are provided by XHR, but are a little tricky to fit in the Promise-based nature of the Fetch API.</p>
<p>You can contribute to the evolution of this API by participating in discussions on the <a href="https://whatwg.org/mailing-list">WHATWG mailing list</a> and in the issues in the <a href="https://www.w3.org/Bugs/Public/buglist.cgi?product=WHATWG&amp;component=Fetch&amp;resolution=---">Fetch</a> and <a href="https://github.com/slightlyoff/ServiceWorker/issues">ServiceWorker</a>specifications.</p>
<p>For a better web!</p>
<p><em>The author would like to thank Andrea Marchesini, Anne van Kesteren and Ben<br/>
Kelly for helping with the specification and implementation.</em> </p>
</article>
</div>
</div>

6
test/test-pages/bug-1255978/expected.html
Unescape Escape View File

@ -5,7 +5,6 @@
<p>Zeev Sharon and Michael Forrest Jones both run hotel start-ups in the US. Forrest Jones runs the start-up Beechmont Hotels Corporation, a hotel operating company that consults with hotel owners on how they can improve their business. Sharon is the CEO of Hotelied, a start-up that allows people to sign up for discounts at luxury hotels.</p>
<p>But even luxury hotels aren’t always cleaned as often as they should be.</p>
<p>Here are some of the secrets that the receptionist will never tell you when you check in, according to answers posted on <a href="https://www.quora.com/What-are-the-things-we-dont-know-about-hotel-rooms" target="_blank">Quora</a>.</p>
<h3> </h3>
<div class="dnd-widget-wrapper context-sdl_editor_representation type-image">
<div class="dnd-atom-rendered">
<div class="image"><img src="https://static.independent.co.uk/s3fs-public/styles/story_medium/public/thumbnails/image/2014/03/18/10/bandb2.jpg" alt="bandb2.jpg" title="bandb2.jpg" width="564" height="423" /></div>
@ -17,7 +16,6 @@
<div class="dnd-widget-wrapper context-sdl_editor_representation type-video">
<p class="dnd-caption-wrapper">Video shows bed bug infestation at New York hotel</p>
</div>
<h3> </h3>
<div class="dnd-widget-wrapper context-sdl_editor_representation type-image">
<div class="dnd-atom-rendered">
<div class="image"><img src="https://static.independent.co.uk/s3fs-public/styles/story_medium/public/thumbnails/image/2015/05/26/11/hotel-door-getty.jpg" alt="hotel-door-getty.jpg" title="hotel-door-getty.jpg" width="564" height="423" /></div>
@ -26,7 +24,6 @@
</div>
<p>2. Check the peep hole has not been tampered with</p>
<p>This is not common, but can happen, Forrest Jones said. He advised stuffing the peep hole with a strip of rolled up notepaper when not in use. When someone knocks on the door, the paper can be removed to check who is there. If no one is visible, he recommends calling the front desk immediately. “I look forward to the day when I can tell you to choose only hotels where every employee who has access to guestroom keys is subjected to a complete public records background check, prior to hire, and every year or two thereafter. But for now, I can't,” he said.</p>
<h3> </h3>
<div class="dnd-widget-wrapper context-sdl_editor_representation type-image">
<div class="dnd-atom-rendered">
<div class="image"><img src="https://static.independent.co.uk/s3fs-public/styles/story_medium/public/thumbnails/image/2013/07/31/15/luggage-3.jpg" alt="luggage-3.jpg" title="luggage-3.jpg" width="564" height="423" /></div>
@ -35,7 +32,6 @@
</div>
<p>3. Don’t use a wooden luggage rack</p>
<p>Bedbugs love wood. Even though a wooden luggage rack might look nicer and more expensive than a metal one, it’s a breeding ground for bugs. Forrest Jones says guests should put the items they plan to take from bags on other pieces of furniture and leave the bag on the floor.</p>
<h3> </h3>
<div class="dnd-widget-wrapper context-sdl_editor_representation type-image">
<div class="dnd-atom-rendered">
<div class="image"><img src="https://static.independent.co.uk/s3fs-public/styles/story_medium/public/thumbnails/image/2015/04/13/11/Lifestyle-hotels.jpg" alt="Lifestyle-hotels.jpg" title="Lifestyle-hotels.jpg" width="564" height="423" /></div>
@ -46,7 +42,6 @@
<p>Zeev Sharon said that the old rule of thumb is that for every $1000 invested in a room, the hotel should charge $1 in average daily rate. So a room that cost $300,000 to build, should sell on average for $300/night.</p>
<h3>5. Beware the wall-mounted hairdryer</h3>
<p>It contains the most germs of anything in the room. Other studies have said the TV remote and bedside lamp switches are the most unhygienic. “Perhaps because it's something that's easy for the housekeepers to forget to check or to squirt down with disinfectant,” Forrest Jones said.</p>
<h3> </h3>
<div class="dnd-widget-wrapper context-full type-gallery">
<div class="dnd-atom-rendered">
<div class="container grid-mod-gallery" data-scald-gallery="3739501">
@ -56,7 +51,6 @@
</div>
<h3>6. Mini bars almost always lose money</h3>
<p>Despite the snacks in the minibar seeming like the most overpriced food you have ever seen, hotel owners are still struggling to make a profit from those snacks. "Minibars almost always lose money, even when they charge $10 for a Diet Coke,” Sharon said.</p>
<h3> </h3>
<div class="dnd-widget-wrapper context-sdl_editor_representation type-image">
<div class="dnd-atom-rendered">
<div class="image"><img src="https://static.independent.co.uk/s3fs-public/styles/story_medium/public/thumbnails/image/2014/03/13/16/agenda7.jpg" alt="agenda7.jpg" title="agenda7.jpg" width="564" height="423" /></div>

4
test/test-pages/clean-links/expected.html
Unescape Escape View File

@ -1,6 +1,6 @@
<div id="readability-page-1" class="page">
<div>
<td>
<table width="675 " border="0 " cellpadding="2 "><tbody><tr><td>
<h3 align="center ">Study Webtext</h3>
<h2 align="center "><span face="Lucida Handwriting " color="Maroon
">"Bartleby the Scrivener: A Story of Wall-Street " </span>(1853)&nbsp;<br/>
@ -259,6 +259,6 @@
<p>* * * * * * * *</p>
<p>There would seem little need for proceeding further in this history. Imagination will readily supply the meagre recital of poor Bartleby's interment. But ere parting with the reader, let me say, that if this little narrative has sufficiently interested him, to awaken curiosity as to who Bartleby was, and what manner of life he led prior to the present narrator's making his acquaintance, I can only reply, that in such curiosity I fully share, but am wholly unable to gratify it. Yet here I hardly know whether I should divulge one little item of rumor, which came to my ear a few months after the scrivener's decease. Upon what basis it rested, I could never ascertain; and hence how true it is I cannot now tell. But inasmuch as this vague report has not been without a certain strange suggestive interest to me, however said, it may prove the same with some others; and so I will briefly mention it. The report was this: that Bartleby had been a subordinate clerk in the Dead Letter Office at <a href="http://raven.cc.ukans.edu/%7Ezeke/bartleby/parker.html" target="_blank">Washington</a>, from which he had been suddenly removed by a change in the administration. When I think over this rumor, I cannot adequately express the emotions which seize me. Dead letters! does it not sound like dead men? Conceive a man by nature and misfortune prone to a pallid hopelessness, can any business seem more fitted to heighten it than that of continually handling these dead letters and assorting them for the flames? For by the cart-load they are annually burned. Sometimes from out the folded paper the pale clerk takes a ring:--the bank-note sent in swiftest charity:--he whom it would relieve, nor eats nor hungers any more; pardon for those who died despairing; hope for those who died unhoping; good tidings for those who died stifled by unrelieved calamities. On errands of life, these letters speed to death. </p>
<p> Ah Bartleby! Ah humanity!</p>
</td>
</td></tr></tbody></table>
</div>
</div>

60
test/test-pages/daringfireball-1/expected.html
Unescape Escape View File

@ -1,31 +1,35 @@
<div id="readability-page-1" class="page">
<div class="article">
<p>Daring Fireball is written and produced by John Gruber.</p>
<p>
<a href="http://fakehost/graphics/author/addison-bw.jpg"> <img src="http://fakehost/graphics/author/addison-bw-425.jpg" alt="Photograph of the author."/></a>
<br/><em>Portrait by <a href="http://superbiate.com/inquiries/">George Del Barrio</a></em> </p>
<h2>Mac Apps</h2>
<ul>
<li><a href="http://www.barebones.com/products/bbedit/">BBEdit</a></li>
<li><a href="http://www.flyingmeat.com/acorn/">Acorn</a></li>
<li><a href="http://www.red-sweater.com/marsedit/">MarsEdit</a></li>
<li><a href="http://aged-and-distilled.com/napkin/">Napkin</a></li>
<li><a href="http://www.barebones.com/products/Yojimbo/">Yojimbo</a></li>
<li><a href="http://www.panic.com/transmit/">Transmit</a></li>
<li><a href="http://latenightsw.com/sd4/index.html">Script Debugger</a></li>
<li><a href="http://www.ambrosiasw.com/utilities/snapzprox/">Snapz Pro X</a></li>
<li><a href="http://nightly.webkit.org/">WebKit</a></li>
</ul>
<h2>iPhone Apps</h2>
<ul>
<li><a href="http://vesperapp.co/">Vesper</a></li>
</ul>
<h2>Server Software</h2>
<p>The Daring Fireball website is hosted by <a href="http://joyent.com/">Joyent</a>.</p>
<p>Articles and links are published through <a href="http://movabletype.org/">Movable Type</a>. In addition to my own SmartyPants and Markdown plug-ins, Daring Fireball uses several excellent Movable Type plug-ins, including Brad Choate’s <a href="http://bradchoate.com/weblog/2003/06/24/regular-expressions">MT-Regex</a> and <a href="http://bradchoate.com/weblog/2004/10/20/mtifempty">MT-IfEmpty</a>, and <a href="http://bumppo.net/projects/amputator/">Nat Irons’s Amputator</a>.</p>
<p>Stats are tracked using <a href="http://haveamint.com/">Mint</a>. Additional web nerdery, including the membership system, is fueled by <a href="http://perl.org/">Perl</a>, <a href="http://www.php.net/">PHP</a>, and <a href="http://www.mysql.com/">MySQL</a>.</p>
<h2>Web Standards</h2>
<p>Web standards are important, and Daring Fireball adheres to them. Specifically, Daring Fireball’s HTML markup should validate as either <a href="http://www.whatwg.org/specs/web-apps/current-work/">HTML 5</a> or XHTML 4.01 Transitional, its layout is constructed using <a href="http://jigsaw.w3.org/css-validator/validator?uri=http://daringfireball.net/css/fireball_screen.css">valid CSS</a>, and its syndicated feed is <a href="http://feedvalidator.org/check?url=http%3A%2F%2Fdaringfireball.net%2Findex.xml">valid Atom</a>.</p>
<p>If Daring Fireball looks goofy in your browser, you’re likely using a shitty browser that doesn’t support web standards. Internet Explorer, I’m looking in your direction. If you complain about this, I will laugh at you, because I do not care. If, however, you are using a modern, standards-compliant browser and have trouble viewing or reading Daring Fireball, please do let me know.</p>
<div id="Box">
<div id="Main">
<div class="article">
<p>Daring Fireball is written and produced by John Gruber.</p>
<p>
<a href="http://fakehost/graphics/author/addison-bw.jpg"> <img src="http://fakehost/graphics/author/addison-bw-425.jpg" alt="Photograph of the author."/></a>
<br/><em>Portrait by <a href="http://superbiate.com/inquiries/">George Del Barrio</a></em> </p>
<h2>Mac Apps</h2>
<ul>
<li><a href="http://www.barebones.com/products/bbedit/">BBEdit</a></li>
<li><a href="http://www.flyingmeat.com/acorn/">Acorn</a></li>
<li><a href="http://www.red-sweater.com/marsedit/">MarsEdit</a></li>
<li><a href="http://aged-and-distilled.com/napkin/">Napkin</a></li>
<li><a href="http://www.barebones.com/products/Yojimbo/">Yojimbo</a></li>
<li><a href="http://www.panic.com/transmit/">Transmit</a></li>
<li><a href="http://latenightsw.com/sd4/index.html">Script Debugger</a></li>
<li><a href="http://www.ambrosiasw.com/utilities/snapzprox/">Snapz Pro X</a></li>
<li><a href="http://nightly.webkit.org/">WebKit</a></li>
</ul>
<h2>iPhone Apps</h2>
<ul>
<li><a href="http://vesperapp.co/">Vesper</a></li>
</ul>
<h2>Server Software</h2>
<p>The Daring Fireball website is hosted by <a href="http://joyent.com/">Joyent</a>.</p>
<p>Articles and links are published through <a href="http://movabletype.org/">Movable Type</a>. In addition to my own SmartyPants and Markdown plug-ins, Daring Fireball uses several excellent Movable Type plug-ins, including Brad Choate’s <a href="http://bradchoate.com/weblog/2003/06/24/regular-expressions">MT-Regex</a> and <a href="http://bradchoate.com/weblog/2004/10/20/mtifempty">MT-IfEmpty</a>, and <a href="http://bumppo.net/projects/amputator/">Nat Irons’s Amputator</a>.</p>
<p>Stats are tracked using <a href="http://haveamint.com/">Mint</a>. Additional web nerdery, including the membership system, is fueled by <a href="http://perl.org/">Perl</a>, <a href="http://www.php.net/">PHP</a>, and <a href="http://www.mysql.com/">MySQL</a>.</p>
<h2>Web Standards</h2>
<p>Web standards are important, and Daring Fireball adheres to them. Specifically, Daring Fireball’s HTML markup should validate as either <a href="http://www.whatwg.org/specs/web-apps/current-work/">HTML 5</a> or XHTML 4.01 Transitional, its layout is constructed using <a href="http://jigsaw.w3.org/css-validator/validator?uri=http://daringfireball.net/css/fireball_screen.css">valid CSS</a>, and its syndicated feed is <a href="http://feedvalidator.org/check?url=http%3A%2F%2Fdaringfireball.net%2Findex.xml">valid Atom</a>.</p>
<p>If Daring Fireball looks goofy in your browser, you’re likely using a shitty browser that doesn’t support web standards. Internet Explorer, I’m looking in your direction. If you complain about this, I will laugh at you, because I do not care. If, however, you are using a modern, standards-compliant browser and have trouble viewing or reading Daring Fireball, please do let me know.</p>
</div>
</div>
</div>
</div>

56
test/test-pages/iab-1/expected.html
Unescape Escape View File

@ -1,30 +1,32 @@
<div id="readability-page-1" class="page">
<div class="article__content">
<p>We messed up. As technologists, tasked with delivering content and services to users, we lost track of the user experience.</p>
<p>Twenty years ago we saw an explosion of websites, built by developers around the world, providing all forms of content. This was the beginning of an age of enlightenment, the intersection of content and technology. Many of us in the technical field felt compelled, and even empowered, to produce information as the distribution means for mass communication were no longer restricted by a high barrier to entry.</p>
<p>In 2000, the dark ages came when the dot-com bubble burst. We were told that our startups were gone or that our divisions sustained by corporate parent companies needed to be in the black. It was a wakeup call that led to a renaissance age. Digital advertising became the foundation of an economic engine that, still now, sustains the free and democratic World Wide Web. In digital publishing, we strived to balance content, commerce, and technology. The content management systems and communication gateways we built to inform and entertain populations around the world disrupted markets and in some cases governments, informed communities of imminent danger, and liberated new forms of art and entertainment—all while creating a digital middle class of small businesses.</p>
<p>We engineered not just the technical, but also the social and economic foundation that users around the world came to lean on for access to real time information. And users came to expect this information whenever and wherever they needed it. And more often than not, for anybody with a connected device, it was free.</p>
<p>This was choice—powered by digital advertising—and premised on user experience.</p>
<p>But we messed up.</p>
<p>Through our pursuit of further automation and maximization of margins during the industrial age of media technology, we built advertising technology to optimize publishers’ yield of marketing budgets that had eroded after the last recession. Looking back now, our scraping of dimes may have cost us dollars in consumer loyalty. The fast, scalable systems of targeting users with ever-heftier advertisements have slowed down the public internet and drained more than a few batteries. We were so clever and so good at it that we over-engineered the capabilities of the plumbing laid down by, well, ourselves. This steamrolled the users, depleted their devices, and tried their patience.</p>
<p>The rise of ad blocking poses a threat to the internet and could potentially drive users to an enclosed platform world dominated by a few companies. We have let the fine equilibrium of content, commerce, and technology get out of balance in the open web. We had, and still do have, a responsibility to educate the business side, and in some cases to push back. We lost sight of our social and ethical responsibility to provide a safe, usable experience for anyone and everyone wanting to consume the content of their choice.</p>
<p>We need to bring that back into alignment, starting right now.</p>
<p>
<a href="http://www.iab.com/wp-content/uploads/2015/10/getting-lean-with-digital-ad-ux.jpg"><img width="300" height="250" alt="Getting LEAN with Digital Ad UX" src="http://www.iab.com/wp-content/uploads/2015/10/getting-lean-with-digital-ad-ux-300x250.jpg" class="alignnone size-medium wp-image-15403"/></a>Today, the IAB Tech Lab is launching the L.E.A.N. Ads program. Supported by the Executive Committee of the IAB Tech Lab Board, IABs around the world, and hundreds of member companies, L.E.A.N. stands for Light, Encrypted, Ad choice supported, Non-invasive ads. These are principles that will help guide the next phases of advertising technical standards for the global digital advertising supply chain.</p>
<p>As with any other industry, standards should be created by non-profit standards-setting bodies, with many diverse voices providing input. We will invite all parties for public comment, and make sure consumer interest groups have the opportunity to provide input.</p>
<p>L.E.A.N. Ads do not replace the current advertising standards many consumers still enjoy and engage with while consuming content on our sites across all IP enabled devices. Rather, these principles will guide an alternative set of standards that provide choice for marketers, content providers, and consumers.</p>
<p>Among the many areas of concentration, we must also address frequency capping on retargeting in Ad Tech and make sure a user is targeted appropriately before, but never AFTER they make a purchase. If we are so good at reach and scale, we can be just as good, if not better, at moderation. Additionally, we must address volume of ads per page as well as continue on the path to viewability. The dependencies here are critical to an optimized user experience.</p>
<p>The consumer is demanding these actions, challenging us to do better, and we must respond.</p>
<p>The IAB Tech Lab will continue to provide the tools for publishers in the digital supply chain to have a dialogue with users about their choices so that content providers can generate revenue while creating value. Publishers should have the opportunity to provide rich advertising experiences, L.E.A.N. advertising experiences, and subscription services. Or publishers can simply deny their service to users who choose to keep on blocking ads. That is all part of elasticity of consumer tolerance and choice.</p>
<p>Finally, we must do this in an increasingly fragmented market, across screens. We must do this in environments where entire sites are blocked, purposefully or not. Yes, it is disappointing that our development efforts will have to manage with multiple frameworks while we work to supply the economic engine to sustain an open internet. However, our goal is still to provide diverse content and voices to as many connected users as possible around the world.</p>
<p>That is user experience.</p>
<table>
<tbody>
<tr>
<td>IAB Tech Lab Members can join the IAB Tech Lab Ad Blocking Working Group, please email <a href="mailto:adblocking@iab.com">adblocking@iab.com</a> for more information.</td>
</tr>
</tbody>
</table>
<p>Read <a target="_blank" href="http://www.iab.com/insights/ad-blocking/">more about ad blocking here</a>.</p>
<div class="container module--news">
<div class="article__content">
<p>We messed up. As technologists, tasked with delivering content and services to users, we lost track of the user experience.</p>
<p>Twenty years ago we saw an explosion of websites, built by developers around the world, providing all forms of content. This was the beginning of an age of enlightenment, the intersection of content and technology. Many of us in the technical field felt compelled, and even empowered, to produce information as the distribution means for mass communication were no longer restricted by a high barrier to entry.</p>
<p>In 2000, the dark ages came when the dot-com bubble burst. We were told that our startups were gone or that our divisions sustained by corporate parent companies needed to be in the black. It was a wakeup call that led to a renaissance age. Digital advertising became the foundation of an economic engine that, still now, sustains the free and democratic World Wide Web. In digital publishing, we strived to balance content, commerce, and technology. The content management systems and communication gateways we built to inform and entertain populations around the world disrupted markets and in some cases governments, informed communities of imminent danger, and liberated new forms of art and entertainment—all while creating a digital middle class of small businesses.</p>
<p>We engineered not just the technical, but also the social and economic foundation that users around the world came to lean on for access to real time information. And users came to expect this information whenever and wherever they needed it. And more often than not, for anybody with a connected device, it was free.</p>
<p>This was choice—powered by digital advertising—and premised on user experience.</p>
<p>But we messed up.</p>
<p>Through our pursuit of further automation and maximization of margins during the industrial age of media technology, we built advertising technology to optimize publishers’ yield of marketing budgets that had eroded after the last recession. Looking back now, our scraping of dimes may have cost us dollars in consumer loyalty. The fast, scalable systems of targeting users with ever-heftier advertisements have slowed down the public internet and drained more than a few batteries. We were so clever and so good at it that we over-engineered the capabilities of the plumbing laid down by, well, ourselves. This steamrolled the users, depleted their devices, and tried their patience.</p>
<p>The rise of ad blocking poses a threat to the internet and could potentially drive users to an enclosed platform world dominated by a few companies. We have let the fine equilibrium of content, commerce, and technology get out of balance in the open web. We had, and still do have, a responsibility to educate the business side, and in some cases to push back. We lost sight of our social and ethical responsibility to provide a safe, usable experience for anyone and everyone wanting to consume the content of their choice.</p>
<p>We need to bring that back into alignment, starting right now.</p>
<p>
<a href="http://www.iab.com/wp-content/uploads/2015/10/getting-lean-with-digital-ad-ux.jpg"><img width="300" height="250" alt="Getting LEAN with Digital Ad UX" src="http://www.iab.com/wp-content/uploads/2015/10/getting-lean-with-digital-ad-ux-300x250.jpg" class="alignnone size-medium wp-image-15403"/></a>Today, the IAB Tech Lab is launching the L.E.A.N. Ads program. Supported by the Executive Committee of the IAB Tech Lab Board, IABs around the world, and hundreds of member companies, L.E.A.N. stands for Light, Encrypted, Ad choice supported, Non-invasive ads. These are principles that will help guide the next phases of advertising technical standards for the global digital advertising supply chain.</p>
<p>As with any other industry, standards should be created by non-profit standards-setting bodies, with many diverse voices providing input. We will invite all parties for public comment, and make sure consumer interest groups have the opportunity to provide input.</p>
<p>L.E.A.N. Ads do not replace the current advertising standards many consumers still enjoy and engage with while consuming content on our sites across all IP enabled devices. Rather, these principles will guide an alternative set of standards that provide choice for marketers, content providers, and consumers.</p>
<p>Among the many areas of concentration, we must also address frequency capping on retargeting in Ad Tech and make sure a user is targeted appropriately before, but never AFTER they make a purchase. If we are so good at reach and scale, we can be just as good, if not better, at moderation. Additionally, we must address volume of ads per page as well as continue on the path to viewability. The dependencies here are critical to an optimized user experience.</p>
<p>The consumer is demanding these actions, challenging us to do better, and we must respond.</p>
<p>The IAB Tech Lab will continue to provide the tools for publishers in the digital supply chain to have a dialogue with users about their choices so that content providers can generate revenue while creating value. Publishers should have the opportunity to provide rich advertising experiences, L.E.A.N. advertising experiences, and subscription services. Or publishers can simply deny their service to users who choose to keep on blocking ads. That is all part of elasticity of consumer tolerance and choice.</p>
<p>Finally, we must do this in an increasingly fragmented market, across screens. We must do this in environments where entire sites are blocked, purposefully or not. Yes, it is disappointing that our development efforts will have to manage with multiple frameworks while we work to supply the economic engine to sustain an open internet. However, our goal is still to provide diverse content and voices to as many connected users as possible around the world.</p>
<p>That is user experience.</p>
<table>
<tbody>
<tr>
<td>IAB Tech Lab Members can join the IAB Tech Lab Ad Blocking Working Group, please email <a href="mailto:adblocking@iab.com">adblocking@iab.com</a> for more information.</td>
</tr>
</tbody>
</table>
<p>Read <a target="_blank" href="http://www.iab.com/insights/ad-blocking/">more about ad blocking here</a>.</p>
</div>
</div>
</div>

316
test/test-pages/keep-images/expected.html
Unescape Escape View File

@ -1,160 +1,162 @@
<div id="readability-page-1" class="page">
<div class="postField postField--body">
<section name="ef8c" class=" section--first section--last">
<div class="section-content">
<div class="section-inner u-sizeFullWidth">
<figure name="b9ad" id="b9ad" class="graf--figure postField--fillWidthImage graf--first">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*sLDnS1UWEFIS33uLMxq3cw.jpeg" data-width="2100" data-height="1402" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*sLDnS1UWEFIS33uLMxq3cw.jpeg"/></div>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<h4 name="9736" id="9736" data-align="center" class="graf--h4">Welcome to DoctorX’s Barcelona lab, where the drugs you bought online are tested for safety and purity. No questions asked.</h4>
<figure name="7417" id="7417" class="graf--figure">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*3vIhkoHIzcxvUdijoCVx6w.png" data-width="1200" data-height="24" data-action="zoom" data-action-value="1*3vIhkoHIzcxvUdijoCVx6w.png" src="https://d262ilb51hltx0.cloudfront.net/max/800/1*3vIhkoHIzcxvUdijoCVx6w.png"/></div>
</figure>
<p name="8a83" id="8a83" class="graf--p">Standing at a table in a chemistry lab in Barcelona, Cristina Gil Lladanosa tears open a silver, smell-proof protective envelope. She slides out a transparent bag full of crystals. Around her, machines whir and hum, and other researchers mill around in long, white coats.</p>
<p name="b675" id="b675" class="graf--p">She is holding the lab’s latest delivery of a drug bought from the “deep web,” the clandestine corner of the internet that isn’t reachable by normal search engines, and is home to some sites that require special software to access. Labeled as <a href="http://en.wikipedia.org/wiki/MDMA" data-href="http://en.wikipedia.org/wiki/MDMA" class="markup--anchor markup--p-anchor" rel="nofollow">MDMA</a> (the street term is ecstasy), this sample has been shipped from Canada. Lladanosa and her colleague Iván Fornís Espinosa have also received drugs, anonymously, from people in China, Australia, Europe and the United States.</p>
<p name="3c0b" id="3c0b" class="graf--p graf--startsWithDoubleQuote">“Here we have speed, MDMA, cocaine, pills,” Lladanosa says, pointing to vials full of red, green, blue and clear solutions sitting in labeled boxes.</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="c4e6" id="c4e6" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*4gN1-fzOwCniw-DbqQjDeQ.jpeg" data-width="2100" data-height="1402" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*4gN1-fzOwCniw-DbqQjDeQ.jpeg"/></div>
<figcaption class="imageCaption">Cristina Gil Lladanosa, at the Barcelona testing lab | photo by Joan Bardeletti</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<p name="7a54" id="7a54" class="graf--p">Since 2011, with the launch of <a href="http://en.wikipedia.org/wiki/Silk_Road_%28marketplace%29" data-href="http://en.wikipedia.org/wiki/Silk_Road_%28marketplace%29" class="markup--anchor markup--p-anchor" rel="nofollow">Silk Road</a>, anybody has been able to safely buy illegal drugs from the deep web and have them delivered to their door. Though the FBI shut down that black market in October 2013, other outlets have emerged to fill its role. For the last 10 months the lab at which Lladanosa and Espinosa work has offered a paid testing service of those drugs. By sending in samples for analysis, users can know exactly what it is they are buying, and make a more informed decision about whether to ingest the substance. The group, called <a href="http://energycontrol.org/" data-href="http://energycontrol.org/" class="markup--anchor markup--p-anchor" rel="nofollow">Energy Control</a>, which has being running “harm reduction” programs since 1999, is the first to run a testing service explicitly geared towards verifying those purchases from the deep web.</p>
<p name="4395" id="4395" class="graf--p">Before joining Energy Control, Lladanosa briefly worked at a pharmacy, whereas Espinosa spent 14 years doing drug analysis. Working at Energy Control is “more gratifying,” and “rewarding” than her previous jobs, Lladanosa told me. They also receive help from a group of volunteers, made up of a mixture of “squatters,” as Espinosa put it, and medical students, who prepare the samples for testing.</p>
<p name="0c18" id="0c18" class="graf--p">After weighing out the crystals, aggressively mixing it with methanol until dissolved, and delicately pouring the liquid into a tiny brown bottle, Lladanosa, a petite woman who is nearly engulfed by her lab coat, is now ready to test the sample. She loads a series of three trays on top of a large white appliance sitting on a table, called a gas chromatograph (GC). A jungle of thick pipes hang from the lab’s ceiling behind it.</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="559c" id="559c" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*2KPmZkIBUrhps-2uwDvYFQ.jpeg" data-width="2100" data-height="1402" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*2KPmZkIBUrhps-2uwDvYFQ.jpeg"/></div>
<figcaption class="imageCaption">Photo by Joan Bardeletti</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<p name="1549" id="1549" class="graf--p graf--startsWithDoubleQuote">“Chromatography separates all the substances,” Lladanosa says as she loads the machine with an array of drugs sent from the deep web and local Spanish users. It can tell whether a sample is pure or contaminated, and if the latter, with what.</p>
<p name="5d0f" id="5d0f" class="graf--p">Rushes of hot air blow across the desk as the gas chromatograph blasts the sample at 280 degrees Celsius. Thirty minutes later the machine’s robotic arm automatically moves over to grip another bottle. The machine will continue cranking through the 150 samples in the trays for most of the work week.</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="d6aa" id="d6aa" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*PU40bbbox2Ompc5I3RE99A.jpeg" data-width="2013" data-height="1241" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*PU40bbbox2Ompc5I3RE99A.jpeg"/></div>
<figcaption class="imageCaption">Photo by Joan Bardeletti</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<p name="15e0" id="15e0" class="graf--p">To get the drugs to Barcelona, a user mails at least 10 milligrams of a substance to the offices of the Asociación Bienestar y Desarrollo, the non-government organization that oversees Energy Control. The sample then gets delivered to the testing service’s laboratory, at the Barcelona Biomedical Research Park, a futuristic, seven story building sitting metres away from the beach. Energy Control borrows its lab space from a biomedical research group for free.</p>
<p name="2574" id="2574" class="graf--p">The tests cost 50 Euro per sample. Users pay, not surprisingly, with Bitcoin. In the post announcing Energy Control’s service on the deep web, the group promised that “All profits of this service are set aside of maintenance of this project.”</p>
<p name="2644" id="2644" class="graf--p">About a week after testing, those results are sent in a PDF to an email address provided by the anonymous client.</p>
<p name="9f91" id="9f91" class="graf--p graf--startsWithDoubleQuote">“The process is quite boring, because you are in a routine,” Lladanosa says. But one part of the process is consistently surprising: that moment when the results pop up on the screen. “Every time it’s something different.” For instance, one cocaine sample she had tested also contained phenacetin, a painkiller added to increase the product’s weight; lidocaine, an anesthetic that numbs the gums, giving the impression that the user is taking higher quality cocaine; and common caffeine.</p>
<figure name="b821" id="b821" class="graf--figure">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*ohyycinH18fz98TCyUzVgQ.png" data-width="1200" data-height="24" data-action="zoom" data-action-value="1*ohyycinH18fz98TCyUzVgQ.png" src="https://d262ilb51hltx0.cloudfront.net/max/800/1*ohyycinH18fz98TCyUzVgQ.png"/></div>
</figure>
<p name="39a6" id="39a6" class="graf--p">The deep web drug lab is the brainchild of Fernando Caudevilla, a Spanish physician who is better known as “DoctorX” on the deep web, a nickname given to him by his Energy Control co-workers because of his earlier writing about the history, risks and recreational culture of MDMA. In the physical world, Caudevilla has worked for over a decade with Energy Control on various harm reduction focused projects, most of which have involved giving Spanish illegal drug users medical guidance, and often writing leaflets about the harms of certain substances.</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="eebc" id="eebc" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*mKvUNOAVQxl6atCbxbCZsg.jpeg" data-width="2100" data-height="1241" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*mKvUNOAVQxl6atCbxbCZsg.jpeg"/></div>
<figcaption class="imageCaption">Fernando Caudevilla, AKA DoctorX. Photo: Joseph Cox</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<p name="c099" id="c099" class="graf--p">Caudevilla first ventured into Silk Road forums in April 2013. “I would like to contribute to this forum offering professional advice in topics related to drug use and health,” he wrote in an <a href="http://web.archive.org/web/20131015051405/https://dkn255hz262ypmii.onion.to/index.php?topic=147607.0" data-href="http://web.archive.org/web/20131015051405/https://dkn255hz262ypmii.onion.to/index.php?topic=147607.0" class="markup--anchor markup--p-anchor" rel="nofollow">introductory post</a>, using his DoctorX alias. Caudevilla offered to provide answers to questions that a typical doctor is not prepared, or willing, to respond to, at least not without a lecture or a judgment. “This advice cannot replace a complete face-to-face medical evaluation,” he wrote, “but I know how difficult it can be to talk frankly about these things.”</p>
<p name="ff1d" id="ff1d" class="graf--p">The requests flooded in. A diabetic asked what effect MDMA has on blood sugar; another what the risks of frequent psychedelic use were for a young person. Someone wanted to know whether amphetamine use should be avoided during lactation. In all, Fernando’s thread received over 50,000 visits and 300 questions before the FBI shut down Silk Road.</p>
<p name="1f35" id="1f35" class="graf--p graf--startsWithDoubleQuote">“He’s amazing. A gift to this community,” one user wrote on the Silk Road 2.0 forum, a site that sprang up after the original. “His knowledge is invaluable, and never comes with any judgment.” Up until recently, Caudevilla answered questions on the marketplace “Evolution.” Last week, however, the administrators of that site <a href="http://motherboard.vice.com/read/one-of-the-darknets-biggest-markets-may-have-just-stole-all-its-users-bitcoin" data-href="http://motherboard.vice.com/read/one-of-the-darknets-biggest-markets-may-have-just-stole-all-its-users-bitcoin" class="markup--anchor markup--p-anchor" rel="nofollow">pulled a scam</a>, shutting the market down and escaping with an estimated $12 million worth of Bitcoin.</p>
<p name="b20f" id="b20f" class="graf--p">Caudevilla’s transition from dispensing advice to starting up a no-questions-asked drug testing service came as a consequence of his experience on the deep web. He’d wondered whether he could help bring more harm reduction services to a marketplace without controls. The Energy Control project, as part of its mandate of educating drug users and preventing harm, had already been carrying out drug testing for local Spanish users since 2001, at music festivals, night clubs, or through a drop-in service at a lab in Madrid.</p>
<p name="f739" id="f739" class="graf--p graf--startsWithDoubleQuote">“I thought, we are doing this in Spain, why don’t we do an international drug testing service?” Caudevilla told me when I visited the other Energy Control lab, in Madrid. Caudevilla, a stocky character with ear piercings and short, shaved hair, has eyes that light up whenever he discusses the world of the deep web. Later, via email, he elaborated that it was not a hard sell. “It was not too hard to convince them,” he wrote me. Clearly, Energy Control believed that the reputation he had earned as an unbiased medical professional on the deep web might carry over to the drug analysis service, where one needs to establish “credibility, trustworthiness, [and] transparency,” Caudevilla said. “We could not make mistakes,” he added.</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="4058" id="4058" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*knT10_FNVUmqQIBLnutmzQ.jpeg" data-width="4400" data-height="3141" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*knT10_FNVUmqQIBLnutmzQ.jpeg"/></div>
<figcaption class="imageCaption">Photo: Joseph Cox</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<figure name="818c" id="818c" class="graf--figure">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*ohyycinH18fz98TCyUzVgQ.png" data-width="1200" data-height="24" data-action="zoom" data-action-value="1*ohyycinH18fz98TCyUzVgQ.png" src="https://d262ilb51hltx0.cloudfront.net/max/800/1*ohyycinH18fz98TCyUzVgQ.png"/></div>
</figure>
<p name="7b5e" id="7b5e" class="graf--p">While the Energy Control lab in Madrid lab only tests Spanish drugs from various sources, it is the Barcelona location which vets the substances bought in the shadowy recesses of of the deep web. Caudevilla no longer runs it, having handed it over to his colleague Ana Muñoz. She maintains a presence on the deep web forums, answers questions from potential users, and sends back reports when they are ready.</p>
<p name="0f0e" id="0f0e" class="graf--p">The testing program exists in a legal grey area. The people who own the Barcelona lab are accredited to experiment with and handle drugs, but Energy Control doesn’t have this permission itself, at least not in writing.</p>
<p name="e002" id="e002" class="graf--p graf--startsWithDoubleQuote">“We have a verbal agreement with the police and other authorities. They already know what we are doing,” Lladanosa tells me. It is a pact of mutual benefit. Energy Control provides the police with information on batches of drugs in Spain, whether they’re from the deep web or not, Espinosa says. They also contribute to the European Monitoring Centre for Drugs and Drug Addiction’s early warning system, a collaboration that attempts to spread information about dangerous drugs as quickly as possible.</p>
<p name="db1b" id="db1b" class="graf--p">By the time of my visit in February, Energy Control had received over 150 samples from the deep web and have been receiving more at a rate of between 4 and 8 a week. Traditional drugs, such as cocaine and MDMA, make up about 70 percent of the samples tested, but the Barcelona lab has also received samples of the prescription pill codeine, research chemicals and synthetic cannabinoids, and even pills of Viagra.</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="b885" id="b885" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*Vr61dyCTRwk6CemmVF8YAQ.jpeg" data-width="2100" data-height="1402" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*Vr61dyCTRwk6CemmVF8YAQ.jpeg"/></div>
<figcaption class="imageCaption">Photo by Joan Bardeletti</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<p name="e76f" id="e76f" class="graf--p">So it’s fair to make a tentative judgement on what people are paying for on the deep web. The verdict thus far? Overall, drugs on the deep web appear to be of much higher quality than those found on the street.</p>
<p name="5352" id="5352" class="graf--p graf--startsWithDoubleQuote">“In general, the cocaine is amazing,” says Caudevilla, saying that the samples they’ve seen have purities climbing towards 80 or 90 percent, and some even higher. To get an idea of how unusual this is, take a look at the <a href="http://www.unodc.org/documents/wdr2014/Cocaine_2014_web.pdf" data-href="http://www.unodc.org/documents/wdr2014/Cocaine_2014_web.pdf" class="markup--anchor markup--p-anchor" rel="nofollow">United Nations Office on Drugs and Crime World Drug Report 2014</a>, which reports that the average quality of street cocaine in Spain is just over 40 percent, while in the United Kingdom it is closer to 30 percent.“We have found 100 percent [pure] cocaine,” he adds. “That’s really, really strange. That means that, technically, this cocaine has been purified, with clandestine methods.”</p>
<p name="a71c" id="a71c" class="graf--p">Naturally, identifying vendors who sell this top-of-the-range stuff is one of the reasons that people have sent samples to Energy Control. Caudevilla was keen to stress that, officially, Energy Control’s service “is not intended to be a control of drug quality,” meaning a vetting process for identifying the best sellers, but that is exactly how some people have been using it.</p>
<p name="cb5b" id="cb5b" class="graf--p">As one buyer on the Evolution market, elmo666, wrote to me over the site’s messaging system, “My initial motivations were selfish. My primary motivation was to ensure that I was receiving and continue to receive a high quality product, essentially to keep the vendor honest as far as my interactions with them went.”</p>
<p name="d80d" id="d80d" class="graf--p">Vendors on deep web markets advertise their product just like any other outlet does, using flash sales, gimmicky giveaways and promises of drugs that are superior to those of their competitors. The claims, however, can turn out to be empty: despite the test results that show that deep web cocaine vendors typically sell product that is of a better quality than that found on the street, in plenty of cases, the drugs are nowhere near as pure as advertised.</p>
<p name="36de" id="36de" class="graf--p graf--startsWithDoubleQuote">“You won’t be getting anything CLOSE to what you paid for,” one user complained about the cocaine from ‘Mirkov’, a vendor on Evolution. “He sells 65% not 95%.”</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="8544" id="8544" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*a-1_13xE6_ErQ-QSlz6myw.jpeg" data-width="2100" data-height="1402" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*a-1_13xE6_ErQ-QSlz6myw.jpeg"/></div>
<figcaption class="imageCaption">Photo by Joan Bardeletti</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<figure name="d521" id="d521" class="graf--figure">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*ohyycinH18fz98TCyUzVgQ.png" data-width="1200" data-height="24" data-action="zoom" data-action-value="1*ohyycinH18fz98TCyUzVgQ.png" src="https://d262ilb51hltx0.cloudfront.net/max/800/1*ohyycinH18fz98TCyUzVgQ.png"/></div>
</figure>
<p name="126b" id="126b" class="graf--p">Despite the prevalence of people using the service to gauge the quality of what goes up their nose, many users send samples to Energy Control in the spirit of its original mission: keeping themselves alive and healthy. The worst case scenario from drugs purchased on the deep web is, well the worst case. That was the outcome when <a href="http://www.independent.co.uk/news/uk/crime/teenager-patrick-mcmullen-who-died-while-on-skype-had-bought-drugs-from-silk-road-8942329.html" data-href="http://www.independent.co.uk/news/uk/crime/teenager-patrick-mcmullen-who-died-while-on-skype-had-bought-drugs-from-silk-road-8942329.html" class="markup--anchor markup--p-anchor" rel="nofollow">Patrick McMullen,</a> a 17-year-old Scottish student, ingested half a gram of MDMA and three tabs of LSD, reportedly purchased from the Silk Road. While talking to his friends on Skype, his words became slurred and he passed out. Paramedics could not revive him. The coroner for that case, Sherrif Payne, who deemed the cause of death ecstasy toxicity, told <em class="markup--em markup--p-em">The Independent</em> “You never know the purity of what you are taking and you can easily come unstuck.”</p>
<p name="5e9e" id="5e9e" class="graf--p">ScreamMyName, a deep web user who has been active since the original Silk Road, wants to alert users to the dangerous chemicals that are often mixed with drugs, and is using Energy Control as a means to do so.</p>
<p name="19a6" id="19a6" class="graf--p graf--startsWithDoubleQuote">“We’re at a time where some vendors are outright sending people poison. Some do it unknowingly,” ScreamMyName told me in an encrypted message. “Cocaine production in South America is often tainted with either levamisole or phenacetine. Both poison to humans and both with severe side effects.”</p>
<p name="9fef" id="9fef" class="graf--p">In the case of Levamisole, those prescribing it are often not doctors but veterinarians, as Levamisole is commonly used on animals, primarily for the treatment of worms. If ingested by humans it can lead to cases of extreme eruptions of the skin, as <a href="http://www.ncbi.nlm.nih.gov/pubmed/22127712" data-href="http://www.ncbi.nlm.nih.gov/pubmed/22127712" class="markup--anchor markup--p-anchor" rel="nofollow">documented in a study</a> from researchers at the University of California, San Francisco. But Lladanosa has found Levamisole in cocaine samples; dealers use it to increase the product weight, allowing them to stretch their batch further for greater profit — and also, she says, because Levamisole has a strong stimulant effect.</p>
<p name="7886" id="7886" class="graf--p graf--startsWithDoubleQuote">“It got me sick as fuck,” Dr. Feel, an Evolution user, wrote on the site’s forums after consuming cocaine that had been cut with 23 percent Levamisole, and later tested by Energy Control. “I was laid up in bed for several days because of that shit. The first night I did it, I thought I was going to die. I nearly drove myself to the ER.”</p>
<p name="18d3" id="18d3" class="graf--p graf--startsWithDoubleQuote">“More people die because of tainted drugs than the drugs themselves,” Dr. Feel added. “It’s the cuts and adulterants that are making people sick and killing them.”</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="552a" id="552a" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*IWXhtSsVv0gNnCwnDEXk-Q.jpeg" data-width="2100" data-height="1192" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*IWXhtSsVv0gNnCwnDEXk-Q.jpeg"/></div>
<figcaption class="imageCaption">Photo by Joan Bardeletti</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<p name="839a" id="839a" class="graf--p">The particular case of cocaine cut with Levamisole is one of the reasons that ScreamMyName has been pushing for more drug testing on the deep web markets. “I recognize that drug use isn’t exactly healthy, but why exacerbate the problem?” he told me when I contacted him after his post. “[Energy Control] provides a way for users to test the drugs they’ll use and for these very users to know what it is they’re putting in their bodies. Such services are in very short supply.”</p>
<p name="18dc" id="18dc" class="graf--p">After sending a number of Energy Control tests himself, ScreamMyName started a de facto crowd-sourcing campaign to get more drugs sent to the lab, and then shared the results, after throwing in some cash to get the ball rolling. <a href="https://blockchain.info/address/1Mi6VjMFqjcD48FPV7cnPB24MAtQQenRy3" data-href="https://blockchain.info/address/1Mi6VjMFqjcD48FPV7cnPB24MAtQQenRy3" class="markup--anchor markup--p-anchor" rel="nofollow">He set up a Bitcoin wallet</a>, with the hope that users might chip in to fund further tests. At the time of writing, the wallet has received a total of 1.81 bitcoins; around $430 at today’s exchange rates.</p>
<p name="dcbd" id="dcbd" class="graf--p">In posts to the Evolution community, ScreamMyName pitched this project as something that will benefit users and keep drug dealer honest. “When the funds build up to a point where we can purchase an [Energy Control] test fee, we’ll do a US thread poll for a few days and try to cohesively decide on what vendor to test,” he continued.</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="9d32" id="9d32" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*NGcrjfkV0l37iQH2uyYjEw.jpeg" data-width="1368" data-height="913" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*NGcrjfkV0l37iQH2uyYjEw.jpeg"/></div>
<figcaption class="imageCaption">Photo by Joan Bardeletti</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<p name="bff6" id="bff6" class="graf--p">Other members of the community have been helping out, too. PlutoPete, a vendor from the original Silk Road who sold cannabis seeds and other legal items, has provided ScreamMyName with packaging to safely send the samples to Barcelona. “A box of baggies, and a load of different moisture barrier bags,” PlutoPete told me over the phone. “That’s what all the vendors use.”</p>
<p name="bb78" id="bb78" class="graf--p">It’s a modest program so far. ScreamMyName told me that so far he had gotten enough public funding to purchase five different Energy Control tests, in addition to the ten or so he’s sent himself so far. “The program created is still in its infancy and it is growing and changing as we go along but I have a lot of faith in what we’re doing,” he says.</p>
<p name="5638" id="5638" class="graf--p">But the spirit is contagious: elmo666, the other deep web user testing cocaine, originally kept the results of the drug tests to himself, but he, too, saw a benefit to distributing the data. “It is clear that it is a useful service to other users, keeping vendors honest and drugs (and their users) safe,” he told me. He started to report his findings to others on the forums, and then created a thread with summaries of the test results, as well as comments from the vendors if they provided it. Other users were soon basing their decisions on what to buy on elmo666‘s tests.</p>
<p name="de75" id="de75" class="graf--p graf--startsWithDoubleQuote">“I’m defo trying the cola based on the incredibly helpful elmo and his energy control results and recommendations,” wrote user jayk1984. On top of this, elmo666 plans to launch an independent site on the deep web that will collate all of these results, which should act as a resource for users of all the marketplaces.</p>
<p name="6b72" id="6b72" class="graf--p">As word of elmo666's efforts spread, he began getting requests from drug dealers who wanted him to use their wares for testing. Clearly, they figured that a positive result from Energy Control would be a fantastic marketing tool to draw more customers. They even offered elmo666 free samples. (He passed.)</p>
<p name="b008" id="b008" class="graf--p">Meanwhile, some in the purchasing community are arguing that those running markets on the deep web should be providing quality control themselves. PlutoPete told me over the phone that he had been in discussions about this with Dread Pirate Roberts, the pseudonymous owner of the original Silk Road site. “We [had been] talking about that on a more organized basis on Silk Road 1, doing lots of anonymous buys to police each category. But of course they took the thing [Silk Road] down before we got it properly off the ground,” he lamented.</p>
<p name="49c8" id="49c8" class="graf--p">But perhaps it is best that the users, those who are actually consuming the drugs, remain in charge of shaming dealers and warning each other. “It’s our responsibility to police the market based on reviews and feedback,” elmo666 wrote in an Evolution forum post. It seems that in the lawless space of the deep web, where everything from child porn to weapons are sold openly, users have cooperated in an organic display of self-regulation to stamp out those particular batches of drugs that are more likely to harm users.</p>
<p name="386d" id="386d" class="graf--p graf--startsWithDoubleQuote">“That’s always been the case with the deep web,” PlutoPete told me. Indeed, ever since Silk Road, a stable of the drug markets has been the review system, where buyers can leave a rating and feedback for vendors, letting others know about the reliability of the seller. But DoctorX’s lab, rigorously testing the products with scientific instruments, takes it a step further.</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="890b" id="890b" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*WRlKt3q3mt7utmwxcbl3sQ.jpeg" data-width="2100" data-height="1373" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*WRlKt3q3mt7utmwxcbl3sQ.jpeg"/></div>
<figcaption class="imageCaption">Photo by Joan Bardeletti</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<p name="b109" id="b109" class="graf--p graf--startsWithDoubleQuote">“In the white market, they have quality control. In the dark market, it should be the same,” Cristina Gil Lladanosa says to me before I leave the Barcelona lab.</p>
<p name="e3a4" id="e3a4" class="graf--p">A week after I visit the lab, the results of the MDMA arrive in my inbox: it is 85 percent pure, with no indications of other active ingredients. Whoever ordered that sample from the digital shelves of the deep web, and had it shipped to their doorstep in Canada, got hold of some seriously good, and relatively safe drugs. And now they know it.</p>
<figure name="31cf" id="31cf" class="graf--figure">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*320_4I0lxbn5x3bx4XPI5Q.png" data-width="1200" data-height="24" data-action="zoom" data-action-value="1*320_4I0lxbn5x3bx4XPI5Q.png" src="https://d262ilb51hltx0.cloudfront.net/max/800/1*320_4I0lxbn5x3bx4XPI5Q.png"/></div>
</figure>
<p name="9b87" id="9b87" data-align="center" class="graf--p"><em class="markup--em markup--p-em">Top photo by Joan Bardeletti</em> </p>
<p name="c30a" id="c30a" data-align="center" class="graf--p graf--last">Follow Backchannel: <a href="https://twitter.com/backchnnl" data-href="https://twitter.com/backchnnl" class="markup--anchor markup--p-anchor" rel="nofollow"><em class="markup--em markup--p-em">Twitter</em></a> <em class="markup--em markup--p-em">|</em><a href="https://www.facebook.com/pages/Backchannel/1488568504730671" data-href="https://www.facebook.com/pages/Backchannel/1488568504730671" class="markup--anchor markup--p-anchor" rel="nofollow"><em class="markup--em markup--p-em">Facebook</em></a> </p>
</div>
</div>
</section>
<div class="notesSource">
<div class="postField postField--body">
<section name="ef8c" class=" section--first section--last">
<div class="section-content">
<div class="section-inner u-sizeFullWidth">
<figure name="b9ad" id="b9ad" class="graf--figure postField--fillWidthImage graf--first">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*sLDnS1UWEFIS33uLMxq3cw.jpeg" data-width="2100" data-height="1402" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*sLDnS1UWEFIS33uLMxq3cw.jpeg"/></div>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<h4 name="9736" id="9736" data-align="center" class="graf--h4">Welcome to DoctorX’s Barcelona lab, where the drugs you bought online are tested for safety and purity. No questions asked.</h4>
<figure name="7417" id="7417" class="graf--figure">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*3vIhkoHIzcxvUdijoCVx6w.png" data-width="1200" data-height="24" data-action="zoom" data-action-value="1*3vIhkoHIzcxvUdijoCVx6w.png" src="https://d262ilb51hltx0.cloudfront.net/max/800/1*3vIhkoHIzcxvUdijoCVx6w.png"/></div>
</figure>
<p name="8a83" id="8a83" class="graf--p">Standing at a table in a chemistry lab in Barcelona, Cristina Gil Lladanosa tears open a silver, smell-proof protective envelope. She slides out a transparent bag full of crystals. Around her, machines whir and hum, and other researchers mill around in long, white coats.</p>
<p name="b675" id="b675" class="graf--p">She is holding the lab’s latest delivery of a drug bought from the “deep web,” the clandestine corner of the internet that isn’t reachable by normal search engines, and is home to some sites that require special software to access. Labeled as <a href="http://en.wikipedia.org/wiki/MDMA" data-href="http://en.wikipedia.org/wiki/MDMA" class="markup--anchor markup--p-anchor" rel="nofollow">MDMA</a> (the street term is ecstasy), this sample has been shipped from Canada. Lladanosa and her colleague Iván Fornís Espinosa have also received drugs, anonymously, from people in China, Australia, Europe and the United States.</p>
<p name="3c0b" id="3c0b" class="graf--p graf--startsWithDoubleQuote">“Here we have speed, MDMA, cocaine, pills,” Lladanosa says, pointing to vials full of red, green, blue and clear solutions sitting in labeled boxes.</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="c4e6" id="c4e6" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*4gN1-fzOwCniw-DbqQjDeQ.jpeg" data-width="2100" data-height="1402" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*4gN1-fzOwCniw-DbqQjDeQ.jpeg"/></div>
<figcaption class="imageCaption">Cristina Gil Lladanosa, at the Barcelona testing lab | photo by Joan Bardeletti</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<p name="7a54" id="7a54" class="graf--p">Since 2011, with the launch of <a href="http://en.wikipedia.org/wiki/Silk_Road_%28marketplace%29" data-href="http://en.wikipedia.org/wiki/Silk_Road_%28marketplace%29" class="markup--anchor markup--p-anchor" rel="nofollow">Silk Road</a>, anybody has been able to safely buy illegal drugs from the deep web and have them delivered to their door. Though the FBI shut down that black market in October 2013, other outlets have emerged to fill its role. For the last 10 months the lab at which Lladanosa and Espinosa work has offered a paid testing service of those drugs. By sending in samples for analysis, users can know exactly what it is they are buying, and make a more informed decision about whether to ingest the substance. The group, called <a href="http://energycontrol.org/" data-href="http://energycontrol.org/" class="markup--anchor markup--p-anchor" rel="nofollow">Energy Control</a>, which has being running “harm reduction” programs since 1999, is the first to run a testing service explicitly geared towards verifying those purchases from the deep web.</p>
<p name="4395" id="4395" class="graf--p">Before joining Energy Control, Lladanosa briefly worked at a pharmacy, whereas Espinosa spent 14 years doing drug analysis. Working at Energy Control is “more gratifying,” and “rewarding” than her previous jobs, Lladanosa told me. They also receive help from a group of volunteers, made up of a mixture of “squatters,” as Espinosa put it, and medical students, who prepare the samples for testing.</p>
<p name="0c18" id="0c18" class="graf--p">After weighing out the crystals, aggressively mixing it with methanol until dissolved, and delicately pouring the liquid into a tiny brown bottle, Lladanosa, a petite woman who is nearly engulfed by her lab coat, is now ready to test the sample. She loads a series of three trays on top of a large white appliance sitting on a table, called a gas chromatograph (GC). A jungle of thick pipes hang from the lab’s ceiling behind it.</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="559c" id="559c" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*2KPmZkIBUrhps-2uwDvYFQ.jpeg" data-width="2100" data-height="1402" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*2KPmZkIBUrhps-2uwDvYFQ.jpeg"/></div>
<figcaption class="imageCaption">Photo by Joan Bardeletti</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<p name="1549" id="1549" class="graf--p graf--startsWithDoubleQuote">“Chromatography separates all the substances,” Lladanosa says as she loads the machine with an array of drugs sent from the deep web and local Spanish users. It can tell whether a sample is pure or contaminated, and if the latter, with what.</p>
<p name="5d0f" id="5d0f" class="graf--p">Rushes of hot air blow across the desk as the gas chromatograph blasts the sample at 280 degrees Celsius. Thirty minutes later the machine’s robotic arm automatically moves over to grip another bottle. The machine will continue cranking through the 150 samples in the trays for most of the work week.</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="d6aa" id="d6aa" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*PU40bbbox2Ompc5I3RE99A.jpeg" data-width="2013" data-height="1241" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*PU40bbbox2Ompc5I3RE99A.jpeg"/></div>
<figcaption class="imageCaption">Photo by Joan Bardeletti</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<p name="15e0" id="15e0" class="graf--p">To get the drugs to Barcelona, a user mails at least 10 milligrams of a substance to the offices of the Asociación Bienestar y Desarrollo, the non-government organization that oversees Energy Control. The sample then gets delivered to the testing service’s laboratory, at the Barcelona Biomedical Research Park, a futuristic, seven story building sitting metres away from the beach. Energy Control borrows its lab space from a biomedical research group for free.</p>
<p name="2574" id="2574" class="graf--p">The tests cost 50 Euro per sample. Users pay, not surprisingly, with Bitcoin. In the post announcing Energy Control’s service on the deep web, the group promised that “All profits of this service are set aside of maintenance of this project.”</p>
<p name="2644" id="2644" class="graf--p">About a week after testing, those results are sent in a PDF to an email address provided by the anonymous client.</p>
<p name="9f91" id="9f91" class="graf--p graf--startsWithDoubleQuote">“The process is quite boring, because you are in a routine,” Lladanosa says. But one part of the process is consistently surprising: that moment when the results pop up on the screen. “Every time it’s something different.” For instance, one cocaine sample she had tested also contained phenacetin, a painkiller added to increase the product’s weight; lidocaine, an anesthetic that numbs the gums, giving the impression that the user is taking higher quality cocaine; and common caffeine.</p>
<figure name="b821" id="b821" class="graf--figure">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*ohyycinH18fz98TCyUzVgQ.png" data-width="1200" data-height="24" data-action="zoom" data-action-value="1*ohyycinH18fz98TCyUzVgQ.png" src="https://d262ilb51hltx0.cloudfront.net/max/800/1*ohyycinH18fz98TCyUzVgQ.png"/></div>
</figure>
<p name="39a6" id="39a6" class="graf--p">The deep web drug lab is the brainchild of Fernando Caudevilla, a Spanish physician who is better known as “DoctorX” on the deep web, a nickname given to him by his Energy Control co-workers because of his earlier writing about the history, risks and recreational culture of MDMA. In the physical world, Caudevilla has worked for over a decade with Energy Control on various harm reduction focused projects, most of which have involved giving Spanish illegal drug users medical guidance, and often writing leaflets about the harms of certain substances.</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="eebc" id="eebc" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*mKvUNOAVQxl6atCbxbCZsg.jpeg" data-width="2100" data-height="1241" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*mKvUNOAVQxl6atCbxbCZsg.jpeg"/></div>
<figcaption class="imageCaption">Fernando Caudevilla, AKA DoctorX. Photo: Joseph Cox</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<p name="c099" id="c099" class="graf--p">Caudevilla first ventured into Silk Road forums in April 2013. “I would like to contribute to this forum offering professional advice in topics related to drug use and health,” he wrote in an <a href="http://web.archive.org/web/20131015051405/https://dkn255hz262ypmii.onion.to/index.php?topic=147607.0" data-href="http://web.archive.org/web/20131015051405/https://dkn255hz262ypmii.onion.to/index.php?topic=147607.0" class="markup--anchor markup--p-anchor" rel="nofollow">introductory post</a>, using his DoctorX alias. Caudevilla offered to provide answers to questions that a typical doctor is not prepared, or willing, to respond to, at least not without a lecture or a judgment. “This advice cannot replace a complete face-to-face medical evaluation,” he wrote, “but I know how difficult it can be to talk frankly about these things.”</p>
<p name="ff1d" id="ff1d" class="graf--p">The requests flooded in. A diabetic asked what effect MDMA has on blood sugar; another what the risks of frequent psychedelic use were for a young person. Someone wanted to know whether amphetamine use should be avoided during lactation. In all, Fernando’s thread received over 50,000 visits and 300 questions before the FBI shut down Silk Road.</p>
<p name="1f35" id="1f35" class="graf--p graf--startsWithDoubleQuote">“He’s amazing. A gift to this community,” one user wrote on the Silk Road 2.0 forum, a site that sprang up after the original. “His knowledge is invaluable, and never comes with any judgment.” Up until recently, Caudevilla answered questions on the marketplace “Evolution.” Last week, however, the administrators of that site <a href="http://motherboard.vice.com/read/one-of-the-darknets-biggest-markets-may-have-just-stole-all-its-users-bitcoin" data-href="http://motherboard.vice.com/read/one-of-the-darknets-biggest-markets-may-have-just-stole-all-its-users-bitcoin" class="markup--anchor markup--p-anchor" rel="nofollow">pulled a scam</a>, shutting the market down and escaping with an estimated $12 million worth of Bitcoin.</p>
<p name="b20f" id="b20f" class="graf--p">Caudevilla’s transition from dispensing advice to starting up a no-questions-asked drug testing service came as a consequence of his experience on the deep web. He’d wondered whether he could help bring more harm reduction services to a marketplace without controls. The Energy Control project, as part of its mandate of educating drug users and preventing harm, had already been carrying out drug testing for local Spanish users since 2001, at music festivals, night clubs, or through a drop-in service at a lab in Madrid.</p>
<p name="f739" id="f739" class="graf--p graf--startsWithDoubleQuote">“I thought, we are doing this in Spain, why don’t we do an international drug testing service?” Caudevilla told me when I visited the other Energy Control lab, in Madrid. Caudevilla, a stocky character with ear piercings and short, shaved hair, has eyes that light up whenever he discusses the world of the deep web. Later, via email, he elaborated that it was not a hard sell. “It was not too hard to convince them,” he wrote me. Clearly, Energy Control believed that the reputation he had earned as an unbiased medical professional on the deep web might carry over to the drug analysis service, where one needs to establish “credibility, trustworthiness, [and] transparency,” Caudevilla said. “We could not make mistakes,” he added.</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="4058" id="4058" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*knT10_FNVUmqQIBLnutmzQ.jpeg" data-width="4400" data-height="3141" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*knT10_FNVUmqQIBLnutmzQ.jpeg"/></div>
<figcaption class="imageCaption">Photo: Joseph Cox</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<figure name="818c" id="818c" class="graf--figure">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*ohyycinH18fz98TCyUzVgQ.png" data-width="1200" data-height="24" data-action="zoom" data-action-value="1*ohyycinH18fz98TCyUzVgQ.png" src="https://d262ilb51hltx0.cloudfront.net/max/800/1*ohyycinH18fz98TCyUzVgQ.png"/></div>
</figure>
<p name="7b5e" id="7b5e" class="graf--p">While the Energy Control lab in Madrid lab only tests Spanish drugs from various sources, it is the Barcelona location which vets the substances bought in the shadowy recesses of of the deep web. Caudevilla no longer runs it, having handed it over to his colleague Ana Muñoz. She maintains a presence on the deep web forums, answers questions from potential users, and sends back reports when they are ready.</p>
<p name="0f0e" id="0f0e" class="graf--p">The testing program exists in a legal grey area. The people who own the Barcelona lab are accredited to experiment with and handle drugs, but Energy Control doesn’t have this permission itself, at least not in writing.</p>
<p name="e002" id="e002" class="graf--p graf--startsWithDoubleQuote">“We have a verbal agreement with the police and other authorities. They already know what we are doing,” Lladanosa tells me. It is a pact of mutual benefit. Energy Control provides the police with information on batches of drugs in Spain, whether they’re from the deep web or not, Espinosa says. They also contribute to the European Monitoring Centre for Drugs and Drug Addiction’s early warning system, a collaboration that attempts to spread information about dangerous drugs as quickly as possible.</p>
<p name="db1b" id="db1b" class="graf--p">By the time of my visit in February, Energy Control had received over 150 samples from the deep web and have been receiving more at a rate of between 4 and 8 a week. Traditional drugs, such as cocaine and MDMA, make up about 70 percent of the samples tested, but the Barcelona lab has also received samples of the prescription pill codeine, research chemicals and synthetic cannabinoids, and even pills of Viagra.</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="b885" id="b885" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*Vr61dyCTRwk6CemmVF8YAQ.jpeg" data-width="2100" data-height="1402" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*Vr61dyCTRwk6CemmVF8YAQ.jpeg"/></div>
<figcaption class="imageCaption">Photo by Joan Bardeletti</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<p name="e76f" id="e76f" class="graf--p">So it’s fair to make a tentative judgement on what people are paying for on the deep web. The verdict thus far? Overall, drugs on the deep web appear to be of much higher quality than those found on the street.</p>
<p name="5352" id="5352" class="graf--p graf--startsWithDoubleQuote">“In general, the cocaine is amazing,” says Caudevilla, saying that the samples they’ve seen have purities climbing towards 80 or 90 percent, and some even higher. To get an idea of how unusual this is, take a look at the <a href="http://www.unodc.org/documents/wdr2014/Cocaine_2014_web.pdf" data-href="http://www.unodc.org/documents/wdr2014/Cocaine_2014_web.pdf" class="markup--anchor markup--p-anchor" rel="nofollow">United Nations Office on Drugs and Crime World Drug Report 2014</a>, which reports that the average quality of street cocaine in Spain is just over 40 percent, while in the United Kingdom it is closer to 30 percent.“We have found 100 percent [pure] cocaine,” he adds. “That’s really, really strange. That means that, technically, this cocaine has been purified, with clandestine methods.”</p>
<p name="a71c" id="a71c" class="graf--p">Naturally, identifying vendors who sell this top-of-the-range stuff is one of the reasons that people have sent samples to Energy Control. Caudevilla was keen to stress that, officially, Energy Control’s service “is not intended to be a control of drug quality,” meaning a vetting process for identifying the best sellers, but that is exactly how some people have been using it.</p>
<p name="cb5b" id="cb5b" class="graf--p">As one buyer on the Evolution market, elmo666, wrote to me over the site’s messaging system, “My initial motivations were selfish. My primary motivation was to ensure that I was receiving and continue to receive a high quality product, essentially to keep the vendor honest as far as my interactions with them went.”</p>
<p name="d80d" id="d80d" class="graf--p">Vendors on deep web markets advertise their product just like any other outlet does, using flash sales, gimmicky giveaways and promises of drugs that are superior to those of their competitors. The claims, however, can turn out to be empty: despite the test results that show that deep web cocaine vendors typically sell product that is of a better quality than that found on the street, in plenty of cases, the drugs are nowhere near as pure as advertised.</p>
<p name="36de" id="36de" class="graf--p graf--startsWithDoubleQuote">“You won’t be getting anything CLOSE to what you paid for,” one user complained about the cocaine from ‘Mirkov’, a vendor on Evolution. “He sells 65% not 95%.”</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="8544" id="8544" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*a-1_13xE6_ErQ-QSlz6myw.jpeg" data-width="2100" data-height="1402" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*a-1_13xE6_ErQ-QSlz6myw.jpeg"/></div>
<figcaption class="imageCaption">Photo by Joan Bardeletti</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<figure name="d521" id="d521" class="graf--figure">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*ohyycinH18fz98TCyUzVgQ.png" data-width="1200" data-height="24" data-action="zoom" data-action-value="1*ohyycinH18fz98TCyUzVgQ.png" src="https://d262ilb51hltx0.cloudfront.net/max/800/1*ohyycinH18fz98TCyUzVgQ.png"/></div>
</figure>
<p name="126b" id="126b" class="graf--p">Despite the prevalence of people using the service to gauge the quality of what goes up their nose, many users send samples to Energy Control in the spirit of its original mission: keeping themselves alive and healthy. The worst case scenario from drugs purchased on the deep web is, well the worst case. That was the outcome when <a href="http://www.independent.co.uk/news/uk/crime/teenager-patrick-mcmullen-who-died-while-on-skype-had-bought-drugs-from-silk-road-8942329.html" data-href="http://www.independent.co.uk/news/uk/crime/teenager-patrick-mcmullen-who-died-while-on-skype-had-bought-drugs-from-silk-road-8942329.html" class="markup--anchor markup--p-anchor" rel="nofollow">Patrick McMullen,</a> a 17-year-old Scottish student, ingested half a gram of MDMA and three tabs of LSD, reportedly purchased from the Silk Road. While talking to his friends on Skype, his words became slurred and he passed out. Paramedics could not revive him. The coroner for that case, Sherrif Payne, who deemed the cause of death ecstasy toxicity, told <em class="markup--em markup--p-em">The Independent</em> “You never know the purity of what you are taking and you can easily come unstuck.”</p>
<p name="5e9e" id="5e9e" class="graf--p">ScreamMyName, a deep web user who has been active since the original Silk Road, wants to alert users to the dangerous chemicals that are often mixed with drugs, and is using Energy Control as a means to do so.</p>
<p name="19a6" id="19a6" class="graf--p graf--startsWithDoubleQuote">“We’re at a time where some vendors are outright sending people poison. Some do it unknowingly,” ScreamMyName told me in an encrypted message. “Cocaine production in South America is often tainted with either levamisole or phenacetine. Both poison to humans and both with severe side effects.”</p>
<p name="9fef" id="9fef" class="graf--p">In the case of Levamisole, those prescribing it are often not doctors but veterinarians, as Levamisole is commonly used on animals, primarily for the treatment of worms. If ingested by humans it can lead to cases of extreme eruptions of the skin, as <a href="http://www.ncbi.nlm.nih.gov/pubmed/22127712" data-href="http://www.ncbi.nlm.nih.gov/pubmed/22127712" class="markup--anchor markup--p-anchor" rel="nofollow">documented in a study</a> from researchers at the University of California, San Francisco. But Lladanosa has found Levamisole in cocaine samples; dealers use it to increase the product weight, allowing them to stretch their batch further for greater profit — and also, she says, because Levamisole has a strong stimulant effect.</p>
<p name="7886" id="7886" class="graf--p graf--startsWithDoubleQuote">“It got me sick as fuck,” Dr. Feel, an Evolution user, wrote on the site’s forums after consuming cocaine that had been cut with 23 percent Levamisole, and later tested by Energy Control. “I was laid up in bed for several days because of that shit. The first night I did it, I thought I was going to die. I nearly drove myself to the ER.”</p>
<p name="18d3" id="18d3" class="graf--p graf--startsWithDoubleQuote">“More people die because of tainted drugs than the drugs themselves,” Dr. Feel added. “It’s the cuts and adulterants that are making people sick and killing them.”</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="552a" id="552a" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*IWXhtSsVv0gNnCwnDEXk-Q.jpeg" data-width="2100" data-height="1192" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*IWXhtSsVv0gNnCwnDEXk-Q.jpeg"/></div>
<figcaption class="imageCaption">Photo by Joan Bardeletti</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<p name="839a" id="839a" class="graf--p">The particular case of cocaine cut with Levamisole is one of the reasons that ScreamMyName has been pushing for more drug testing on the deep web markets. “I recognize that drug use isn’t exactly healthy, but why exacerbate the problem?” he told me when I contacted him after his post. “[Energy Control] provides a way for users to test the drugs they’ll use and for these very users to know what it is they’re putting in their bodies. Such services are in very short supply.”</p>
<p name="18dc" id="18dc" class="graf--p">After sending a number of Energy Control tests himself, ScreamMyName started a de facto crowd-sourcing campaign to get more drugs sent to the lab, and then shared the results, after throwing in some cash to get the ball rolling. <a href="https://blockchain.info/address/1Mi6VjMFqjcD48FPV7cnPB24MAtQQenRy3" data-href="https://blockchain.info/address/1Mi6VjMFqjcD48FPV7cnPB24MAtQQenRy3" class="markup--anchor markup--p-anchor" rel="nofollow">He set up a Bitcoin wallet</a>, with the hope that users might chip in to fund further tests. At the time of writing, the wallet has received a total of 1.81 bitcoins; around $430 at today’s exchange rates.</p>
<p name="dcbd" id="dcbd" class="graf--p">In posts to the Evolution community, ScreamMyName pitched this project as something that will benefit users and keep drug dealer honest. “When the funds build up to a point where we can purchase an [Energy Control] test fee, we’ll do a US thread poll for a few days and try to cohesively decide on what vendor to test,” he continued.</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="9d32" id="9d32" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*NGcrjfkV0l37iQH2uyYjEw.jpeg" data-width="1368" data-height="913" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*NGcrjfkV0l37iQH2uyYjEw.jpeg"/></div>
<figcaption class="imageCaption">Photo by Joan Bardeletti</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<p name="bff6" id="bff6" class="graf--p">Other members of the community have been helping out, too. PlutoPete, a vendor from the original Silk Road who sold cannabis seeds and other legal items, has provided ScreamMyName with packaging to safely send the samples to Barcelona. “A box of baggies, and a load of different moisture barrier bags,” PlutoPete told me over the phone. “That’s what all the vendors use.”</p>
<p name="bb78" id="bb78" class="graf--p">It’s a modest program so far. ScreamMyName told me that so far he had gotten enough public funding to purchase five different Energy Control tests, in addition to the ten or so he’s sent himself so far. “The program created is still in its infancy and it is growing and changing as we go along but I have a lot of faith in what we’re doing,” he says.</p>
<p name="5638" id="5638" class="graf--p">But the spirit is contagious: elmo666, the other deep web user testing cocaine, originally kept the results of the drug tests to himself, but he, too, saw a benefit to distributing the data. “It is clear that it is a useful service to other users, keeping vendors honest and drugs (and their users) safe,” he told me. He started to report his findings to others on the forums, and then created a thread with summaries of the test results, as well as comments from the vendors if they provided it. Other users were soon basing their decisions on what to buy on elmo666‘s tests.</p>
<p name="de75" id="de75" class="graf--p graf--startsWithDoubleQuote">“I’m defo trying the cola based on the incredibly helpful elmo and his energy control results and recommendations,” wrote user jayk1984. On top of this, elmo666 plans to launch an independent site on the deep web that will collate all of these results, which should act as a resource for users of all the marketplaces.</p>
<p name="6b72" id="6b72" class="graf--p">As word of elmo666's efforts spread, he began getting requests from drug dealers who wanted him to use their wares for testing. Clearly, they figured that a positive result from Energy Control would be a fantastic marketing tool to draw more customers. They even offered elmo666 free samples. (He passed.)</p>
<p name="b008" id="b008" class="graf--p">Meanwhile, some in the purchasing community are arguing that those running markets on the deep web should be providing quality control themselves. PlutoPete told me over the phone that he had been in discussions about this with Dread Pirate Roberts, the pseudonymous owner of the original Silk Road site. “We [had been] talking about that on a more organized basis on Silk Road 1, doing lots of anonymous buys to police each category. But of course they took the thing [Silk Road] down before we got it properly off the ground,” he lamented.</p>
<p name="49c8" id="49c8" class="graf--p">But perhaps it is best that the users, those who are actually consuming the drugs, remain in charge of shaming dealers and warning each other. “It’s our responsibility to police the market based on reviews and feedback,” elmo666 wrote in an Evolution forum post. It seems that in the lawless space of the deep web, where everything from child porn to weapons are sold openly, users have cooperated in an organic display of self-regulation to stamp out those particular batches of drugs that are more likely to harm users.</p>
<p name="386d" id="386d" class="graf--p graf--startsWithDoubleQuote">“That’s always been the case with the deep web,” PlutoPete told me. Indeed, ever since Silk Road, a stable of the drug markets has been the review system, where buyers can leave a rating and feedback for vendors, letting others know about the reliability of the seller. But DoctorX’s lab, rigorously testing the products with scientific instruments, takes it a step further.</p>
</div>
<div class="section-inner u-sizeFullWidth">
<figure name="890b" id="890b" class="graf--figure postField--fillWidthImage">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*WRlKt3q3mt7utmwxcbl3sQ.jpeg" data-width="2100" data-height="1373" src="https://d262ilb51hltx0.cloudfront.net/max/2000/1*WRlKt3q3mt7utmwxcbl3sQ.jpeg"/></div>
<figcaption class="imageCaption">Photo by Joan Bardeletti</figcaption>
</figure>
</div>
<div class="section-inner layoutSingleColumn">
<p name="b109" id="b109" class="graf--p graf--startsWithDoubleQuote">“In the white market, they have quality control. In the dark market, it should be the same,” Cristina Gil Lladanosa says to me before I leave the Barcelona lab.</p>
<p name="e3a4" id="e3a4" class="graf--p">A week after I visit the lab, the results of the MDMA arrive in my inbox: it is 85 percent pure, with no indications of other active ingredients. Whoever ordered that sample from the digital shelves of the deep web, and had it shipped to their doorstep in Canada, got hold of some seriously good, and relatively safe drugs. And now they know it.</p>
<figure name="31cf" id="31cf" class="graf--figure">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*320_4I0lxbn5x3bx4XPI5Q.png" data-width="1200" data-height="24" data-action="zoom" data-action-value="1*320_4I0lxbn5x3bx4XPI5Q.png" src="https://d262ilb51hltx0.cloudfront.net/max/800/1*320_4I0lxbn5x3bx4XPI5Q.png"/></div>
</figure>
<p name="9b87" id="9b87" data-align="center" class="graf--p"><em class="markup--em markup--p-em">Top photo by Joan Bardeletti</em> </p>
<p name="c30a" id="c30a" data-align="center" class="graf--p graf--last">Follow Backchannel: <a href="https://twitter.com/backchnnl" data-href="https://twitter.com/backchnnl" class="markup--anchor markup--p-anchor" rel="nofollow"><em class="markup--em markup--p-em">Twitter</em></a> <em class="markup--em markup--p-em">|</em><a href="https://www.facebook.com/pages/Backchannel/1488568504730671" data-href="https://www.facebook.com/pages/Backchannel/1488568504730671" class="markup--anchor markup--p-anchor" rel="nofollow"><em class="markup--em markup--p-em">Facebook</em></a> </p>
</div>
</div>
</section>
</div>
</div>
</div>

26
test/test-pages/liberation-1/expected.html
Unescape Escape View File

@ -1,13 +1,17 @@
<div id="readability-page-1" class="page">
<div class="article-body mod" itemprop="articleBody" id="article-body">
<div>
<p>Un troisième Français a été tué dans le tremblement de terre samedi au Népal, emporté par une avalanche, <a href="http://www.liberation.fr/video/2015/04/30/laurent-fabius-plus-de-200-francais-n-ont-pas-ete-retrouves_1278687" target="_blank">a déclaré jeudi le ministre des Affaires étrangères</a>.&nbsp;Les autorités françaises sont toujours sans nouvelles <em>«d’encore plus de 200»&nbsp;</em>personnes.&nbsp;<em>«Pour certains d’entre eux on est très interrogatif»</em>, a ajouté&nbsp;Laurent Fabius. Il accueillait à Roissy un premier avion spécial ramenant des&nbsp;rescapés. <a href="http://www.liberation.fr/video/2015/04/30/seisme-au-nepal-soulages-mais-inquiets-206-survivants-de-retour-en-france_1278758" target="_blank">L’Airbus A350 affrété par les autorités françaises s’est posé peu avant 5h45</a> avec à son bord 206&nbsp;passagers, dont 12&nbsp;enfants et 26&nbsp;blessés, selon une source du Quai d’Orsay. Quasiment tous sont français, à l’exception d’une quinzaine de ressortissants allemands, suisses, italiens, portugais ou encore turcs. Des psychologues, une équipe médicale et des personnels du centre de crise du Quai d’Orsay les attendent.</p>
<p>L’appareil, mis à disposition par Airbus, était arrivé à Katmandou mercredi matin avec 55&nbsp;personnels de santé et humanitaires, ainsi que 25&nbsp;tonnes de matériel (abris, médicaments, aide alimentaire). Un deuxième avion dépêché par Paris, qui était immobilisé aux Emirats depuis mardi avec 20&nbsp;tonnes de matériel, est arrivé jeudi à Katmandou, <a href="http://www.liberation.fr/monde/2015/04/29/embouteillages-et-retards-a-l-aeroport-de-katmandou_1276612" target="_blank">dont le petit aéroport est engorgé</a> par le trafic et l’afflux d’aide humanitaire. Il devait lui aussi ramener des Français, <em>«les plus éprouvés»</em> par la catastrophe et les <em>«plus vulnérables (blessés, familles avec enfants)»</em>, selon le ministère des Affaires étrangères.</p>
<p>2 209 Français ont été localisés sains et saufs tandis que 393 n’ont pas encore pu être joints, selon le Quai d’Orsay. Environ 400&nbsp;Français ont demandé à être rapatriés dans les vols mis en place par la France.</p>
<p>Le séisme a fait près de 5&nbsp;500 morts et touche huit des 28 millions d’habitants du Népal. Des dizaines de milliers de personnes sont sans abri.</p>
<p>
<iframe src="http://www.dailymotion.com/embed/video/x2oikl3" frameborder="0" width="100%" data-aspect-ratio="0.5625" data-responsive="1"></iframe>
<br/><em></em></p>
</div>
</div>
<section id="news-article">
<article itemscope="" itemtype="http://schema.org/NewsArticle">
<div class="article-body mod" itemprop="articleBody" id="article-body">
<div>
<p>Un troisième Français a été tué dans le tremblement de terre samedi au Népal, emporté par une avalanche, <a href="http://www.liberation.fr/video/2015/04/30/laurent-fabius-plus-de-200-francais-n-ont-pas-ete-retrouves_1278687" target="_blank">a déclaré jeudi le ministre des Affaires étrangères</a>.&nbsp;Les autorités françaises sont toujours sans nouvelles <em>«d’encore plus de 200»&nbsp;</em>personnes.&nbsp;<em>«Pour certains d’entre eux on est très interrogatif»</em>, a ajouté&nbsp;Laurent Fabius. Il accueillait à Roissy un premier avion spécial ramenant des&nbsp;rescapés. <a href="http://www.liberation.fr/video/2015/04/30/seisme-au-nepal-soulages-mais-inquiets-206-survivants-de-retour-en-france_1278758" target="_blank">L’Airbus A350 affrété par les autorités françaises s’est posé peu avant 5h45</a> avec à son bord 206&nbsp;passagers, dont 12&nbsp;enfants et 26&nbsp;blessés, selon une source du Quai d’Orsay. Quasiment tous sont français, à l’exception d’une quinzaine de ressortissants allemands, suisses, italiens, portugais ou encore turcs. Des psychologues, une équipe médicale et des personnels du centre de crise du Quai d’Orsay les attendent.</p>
<p>L’appareil, mis à disposition par Airbus, était arrivé à Katmandou mercredi matin avec 55&nbsp;personnels de santé et humanitaires, ainsi que 25&nbsp;tonnes de matériel (abris, médicaments, aide alimentaire). Un deuxième avion dépêché par Paris, qui était immobilisé aux Emirats depuis mardi avec 20&nbsp;tonnes de matériel, est arrivé jeudi à Katmandou, <a href="http://www.liberation.fr/monde/2015/04/29/embouteillages-et-retards-a-l-aeroport-de-katmandou_1276612" target="_blank">dont le petit aéroport est engorgé</a> par le trafic et l’afflux d’aide humanitaire. Il devait lui aussi ramener des Français, <em>«les plus éprouvés»</em> par la catastrophe et les <em>«plus vulnérables (blessés, familles avec enfants)»</em>, selon le ministère des Affaires étrangères.</p>
<p>2 209 Français ont été localisés sains et saufs tandis que 393 n’ont pas encore pu être joints, selon le Quai d’Orsay. Environ 400&nbsp;Français ont demandé à être rapatriés dans les vols mis en place par la France.</p>
<p>Le séisme a fait près de 5&nbsp;500 morts et touche huit des 28 millions d’habitants du Népal. Des dizaines de milliers de personnes sont sans abri.</p>
<p>
<iframe src="http://www.dailymotion.com/embed/video/x2oikl3" frameborder="0" width="100%" data-aspect-ratio="0.5625" data-responsive="1"></iframe>
<br/><em></em></p>
</div>
</div>
</article>
</section>
</div>

56
test/test-pages/medium-2/expected.html
Unescape Escape View File

@ -1,30 +1,32 @@
<div id="readability-page-1" class="page">
<div class="section-inner layoutSingleColumn">
<figure name="4924" id="4924" class="graf--figure graf--first">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*eR_J8DurqygbhrwDg-WPnQ.png" data-width="1891" data-height="1280" data-action="zoom" data-action-value="1*eR_J8DurqygbhrwDg-WPnQ.png" src="https://d262ilb51hltx0.cloudfront.net/max/1600/1*eR_J8DurqygbhrwDg-WPnQ.png"/></div>
<figcaption class="imageCaption">Words need defenders.</figcaption>
</figure>
<h3 name="b098" id="b098" class="graf--h3">On Behalf of “Literally”</h3>
<p name="1a73" id="1a73" class="graf--p">You either are a “literally” abuser or know of one. If you’re anything like me, hearing the word “literally” used incorrectly causes a little piece of your soul to whither and die. Of course I do not mean that literally, I mean that figuratively. An abuser would have said: “Every time a person uses that word, a piece of my soul literally withers and dies.” Which is terribly, horribly wrong.</p>
<p name="104a" id="104a" class="graf--p">For whatever bizarre reason, people feel the need to use literally as a sort of verbal crutch. They use it to emphasize a point, which is silly because they’re already using an analogy or a metaphor to illustrate said point. For example: “Ugh, I literally tore the house apart looking for my remote control!” No, you literally did not tear apart your house, because it’s still standing. If you’d just told me you “tore your house apart” searching for your remote, I would’ve understood what you meant. No need to add “literally” to the sentence.</p>
<p name="c2c0" id="c2c0" class="graf--p">Maybe I should define literally.</p>
<blockquote name="b239" id="b239" class="graf--pullquote pullquote">Literally means actually. When you say something literally happened, you’re describing the scene or situation as it actually happened.</blockquote>
<p name="a8fd" id="a8fd" class="graf--p">So you should only use literally when you mean it. It should not be used in hyperbole. Example: “That was so funny I literally cried.” Which is possible. Some things are funny enough to elicit tears. Note the example stops with “literally cried.” You cannot <em class="markup--em markup--p-em">literally cry your eyes out</em>. The joke wasn’t so funny your eyes popped out of their sockets.</p>
<h4 name="165a" id="165a" class="graf--h4">When in Doubt, Leave it Out</h4>
<p name="e434" id="e434" class="graf--p graf--startsWithDoubleQuote">“I’m so hungry I could eat a horse,” means you’re hungry. You don’t need to say “I’m so hungry I could literally eat a horse.” Because you can’t do that in one sitting, I don’t care how big your stomach is.</p>
<p name="d88f" id="d88f" class="graf--p graf--startsWithDoubleQuote">“That play was so funny I laughed my head off,” illustrates the play was amusing. You don’t need to say you literally laughed your head off, because then your head would be on the ground and you wouldn’t be able to speak, much less laugh.</p>
<p name="4bab" id="4bab" class="graf--p graf--startsWithDoubleQuote">“I drove so fast my car was flying,” we get your point: you were speeding. But your car is never going fast enough to fly, so don’t say your car was literally flying.</p>
<h4 name="f2f0" id="f2f0" class="graf--h4">Insecurities?</h4>
<p name="1bd7" id="1bd7" class="graf--p">Maybe no one believed a story you told as a child, and you felt the need to prove that it actually happened. <em class="markup--em markup--p-em">No really, mom, I literally climbed the tree. </em>In efforts to prove truth, you used literally to describe something real, however outlandish it seemed. Whatever the reason, now your overuse of literally has become a habit.</p>
<h4 name="d7c1" id="d7c1" class="graf--h4">Hard Habit to Break?</h4>
<p name="714b" id="714b" class="graf--p">Abusing literally isn’t as bad a smoking, but it’s still an unhealthy habit (I mean that figuratively). Help is required in order to break it.</p>
<p name="f929" id="f929" class="graf--p">This is my version of an intervention for literally abusers. I’m not sure how else to do it other than in writing. I know this makes me sound like a know-it-all, and I accept that. But there’s no excuse other than blatant ignorance to misuse the word “literally.” So just stop it.</p>
<p name="fd19" id="fd19" class="graf--p">Don’t say “Courtney, this post is so snobbish it literally burned up my computer.” Because nothing is that snobbish that it causes computers to combust. Or: “Courtney, your head is so big it literally cannot get through the door.” Because it can, unless it’s one of those tiny doors from <em class="markup--em markup--p-em">Alice in Wonderland</em> and I need to eat a mushroom to make my whole body smaller.</p>
<h4 name="fe12" id="fe12" class="graf--h4">No One’s Perfect</h4>
<p name="7ff8" id="7ff8" class="graf--p">And I’m not saying I am. I’m trying to restore meaning to a word that’s lost meaning. I’m standing up for literally. It’s a good word when used correctly. People are butchering it and destroying it every day (figuratively speaking) and the massacre needs to stop. Just as there’s a coalition of people against the use of certain fonts (like <a href="http://bancomicsans.com/main/?page_id=2" data-href="http://bancomicsans.com/main/?page_id=2" class="markup--anchor markup--p-anchor" rel="nofollow">Comic Sans</a> and <a href="https://www.facebook.com/group.php?gid=14448723154" data-href="https://www.facebook.com/group.php?gid=14448723154" class="markup--anchor markup--p-anchor" rel="nofollow">Papyrus</a>), so should there be a coalition of people against the abuse of literally.</p>
<h4 name="049e" id="049e" class="graf--h4">Saying it to Irritate?</h4>
<p name="9381" id="9381" class="graf--p">Do you misuse the word “literally” just to annoy your know-it-all or grammar police friends/acquaintances/total strangers? If so, why? Doing so would be like me going outside when it’s freezing, wearing nothing but a pair of shorts and t-shirt in hopes of making you cold by just looking at me. Who suffers more?</p>
<h4 name="3e52" id="3e52" class="graf--h4">Graphical Representation</h4>
<p name="b57e" id="b57e" class="graf--p graf--last">Matthew Inman of “The Oatmeal” wrote a comic about literally. Abusers and defenders alike <a href="http://theoatmeal.com/comics/literally" data-href="http://theoatmeal.com/comics/literally" class="markup--anchor markup--p-anchor" rel="nofollow">should check it out</a>. It’s clear this whole craze about literally is driving a lot of us nuts. You literally abusers are killing off pieces of our souls. You must be stopped, or the world will be lost to meaninglessness forever. Figuratively speaking.</p>
<div class="section-content">
<div class="section-inner layoutSingleColumn">
<figure name="4924" id="4924" class="graf--figure graf--first">
<div class="aspectRatioPlaceholder is-locked"><img class="graf-image" data-image-id="1*eR_J8DurqygbhrwDg-WPnQ.png" data-width="1891" data-height="1280" data-action="zoom" data-action-value="1*eR_J8DurqygbhrwDg-WPnQ.png" src="https://d262ilb51hltx0.cloudfront.net/max/1600/1*eR_J8DurqygbhrwDg-WPnQ.png"/></div>
<figcaption class="imageCaption">Words need defenders.</figcaption>
</figure>
<h3 name="b098" id="b098" class="graf--h3">On Behalf of “Literally”</h3>
<p name="1a73" id="1a73" class="graf--p">You either are a “literally” abuser or know of one. If you’re anything like me, hearing the word “literally” used incorrectly causes a little piece of your soul to whither and die. Of course I do not mean that literally, I mean that figuratively. An abuser would have said: “Every time a person uses that word, a piece of my soul literally withers and dies.” Which is terribly, horribly wrong.</p>
<p name="104a" id="104a" class="graf--p">For whatever bizarre reason, people feel the need to use literally as a sort of verbal crutch. They use it to emphasize a point, which is silly because they’re already using an analogy or a metaphor to illustrate said point. For example: “Ugh, I literally tore the house apart looking for my remote control!” No, you literally did not tear apart your house, because it’s still standing. If you’d just told me you “tore your house apart” searching for your remote, I would’ve understood what you meant. No need to add “literally” to the sentence.</p>
<p name="c2c0" id="c2c0" class="graf--p">Maybe I should define literally.</p>
<blockquote name="b239" id="b239" class="graf--pullquote pullquote">Literally means actually. When you say something literally happened, you’re describing the scene or situation as it actually happened.</blockquote>
<p name="a8fd" id="a8fd" class="graf--p">So you should only use literally when you mean it. It should not be used in hyperbole. Example: “That was so funny I literally cried.” Which is possible. Some things are funny enough to elicit tears. Note the example stops with “literally cried.” You cannot <em class="markup--em markup--p-em">literally cry your eyes out</em>. The joke wasn’t so funny your eyes popped out of their sockets.</p>
<h4 name="165a" id="165a" class="graf--h4">When in Doubt, Leave it Out</h4>
<p name="e434" id="e434" class="graf--p graf--startsWithDoubleQuote">“I’m so hungry I could eat a horse,” means you’re hungry. You don’t need to say “I’m so hungry I could literally eat a horse.” Because you can’t do that in one sitting, I don’t care how big your stomach is.</p>
<p name="d88f" id="d88f" class="graf--p graf--startsWithDoubleQuote">“That play was so funny I laughed my head off,” illustrates the play was amusing. You don’t need to say you literally laughed your head off, because then your head would be on the ground and you wouldn’t be able to speak, much less laugh.</p>
<p name="4bab" id="4bab" class="graf--p graf--startsWithDoubleQuote">“I drove so fast my car was flying,” we get your point: you were speeding. But your car is never going fast enough to fly, so don’t say your car was literally flying.</p>
<h4 name="f2f0" id="f2f0" class="graf--h4">Insecurities?</h4>
<p name="1bd7" id="1bd7" class="graf--p">Maybe no one believed a story you told as a child, and you felt the need to prove that it actually happened. <em class="markup--em markup--p-em">No really, mom, I literally climbed the tree. </em>In efforts to prove truth, you used literally to describe something real, however outlandish it seemed. Whatever the reason, now your overuse of literally has become a habit.</p>
<h4 name="d7c1" id="d7c1" class="graf--h4">Hard Habit to Break?</h4>
<p name="714b" id="714b" class="graf--p">Abusing literally isn’t as bad a smoking, but it’s still an unhealthy habit (I mean that figuratively). Help is required in order to break it.</p>
<p name="f929" id="f929" class="graf--p">This is my version of an intervention for literally abusers. I’m not sure how else to do it other than in writing. I know this makes me sound like a know-it-all, and I accept that. But there’s no excuse other than blatant ignorance to misuse the word “literally.” So just stop it.</p>
<p name="fd19" id="fd19" class="graf--p">Don’t say “Courtney, this post is so snobbish it literally burned up my computer.” Because nothing is that snobbish that it causes computers to combust. Or: “Courtney, your head is so big it literally cannot get through the door.” Because it can, unless it’s one of those tiny doors from <em class="markup--em markup--p-em">Alice in Wonderland</em> and I need to eat a mushroom to make my whole body smaller.</p>
<h4 name="fe12" id="fe12" class="graf--h4">No One’s Perfect</h4>
<p name="7ff8" id="7ff8" class="graf--p">And I’m not saying I am. I’m trying to restore meaning to a word that’s lost meaning. I’m standing up for literally. It’s a good word when used correctly. People are butchering it and destroying it every day (figuratively speaking) and the massacre needs to stop. Just as there’s a coalition of people against the use of certain fonts (like <a href="http://bancomicsans.com/main/?page_id=2" data-href="http://bancomicsans.com/main/?page_id=2" class="markup--anchor markup--p-anchor" rel="nofollow">Comic Sans</a> and <a href="https://www.facebook.com/group.php?gid=14448723154" data-href="https://www.facebook.com/group.php?gid=14448723154" class="markup--anchor markup--p-anchor" rel="nofollow">Papyrus</a>), so should there be a coalition of people against the abuse of literally.</p>
<h4 name="049e" id="049e" class="graf--h4">Saying it to Irritate?</h4>
<p name="9381" id="9381" class="graf--p">Do you misuse the word “literally” just to annoy your know-it-all or grammar police friends/acquaintances/total strangers? If so, why? Doing so would be like me going outside when it’s freezing, wearing nothing but a pair of shorts and t-shirt in hopes of making you cold by just looking at me. Who suffers more?</p>
<h4 name="3e52" id="3e52" class="graf--h4">Graphical Representation</h4>
<p name="b57e" id="b57e" class="graf--p graf--last">Matthew Inman of “The Oatmeal” wrote a comic about literally. Abusers and defenders alike <a href="http://theoatmeal.com/comics/literally" data-href="http://theoatmeal.com/comics/literally" class="markup--anchor markup--p-anchor" rel="nofollow">should check it out</a>. It’s clear this whole craze about literally is driving a lot of us nuts. You literally abusers are killing off pieces of our souls. You must be stopped, or the world will be lost to meaninglessness forever. Figuratively speaking.</p>
</div>
</div>
</div>

48
test/test-pages/qq/expected.html
Unescape Escape View File

@ -1,30 +1,32 @@
<div id="readability-page-1" class="page">
<div id="Cnt-Main-Article-QQ" bosszone="content">
<div class="mbArticleSharePic">
<p class="mbArticleShareBtn"><span>转播到腾讯微博</span></p><img alt="DeepMind新电脑已可利用记忆自学 人工智能迈上新台阶" src="http://img1.gtimg.com/tech/pics/hv1/168/240/2140/139214868.jpg" />
</div>
<p>TNW中文站 10月14日报道</p>
<p><span onmouseover='ShowInfo(this,"GOOG.OQ","200","-1",event);'><a class="a-tips-Article-QQ" href="http://stockhtm.finance.qq.com/astock/ggcx/GOOG.OQ.htm" target="_blank">谷歌</a></span>(<a href="http://t.qq.com/googlechina#pref=qqcom.keyword" rel="googlechina" reltitle="谷歌" target="_blank">微博</a>) 在2014年收购的人工智能公司DeepMind开发出一款能够用自己的记忆学习新知识并利用这些知识来回答问题的计算机。</p>
<p>这款产品具有极其重要的意义，因为这意味着未来的人工智能技术可能不需要人类来教它就能回答人类提出的问题。</p>
<p>DeepMind表示，这款名为DNC（可微神经计算机）的AI模型可以接受家谱和伦敦地铁网络地图这样的信息，还可以回答与那些数据结构中的不同项目之间的关系有关的复杂问题。</p>
<p>例如，它可以回答“从邦德街开始，沿着中央线坐一站，环线坐四站，然后转朱比利线坐两站，你会到达哪个站？”这样的问题。</p>
<p>DeepMind称，DNC还可以帮你规划从沼泽门到皮卡迪利广场的最佳路线。</p>
<p>同样，它还可以理解和回答某个大家族中的成员之间的关系这样的复杂问题，比如“张三的大舅是谁？”。</p>
<p>DNC建立在神经网络的概念之上，神经网络可以模拟人类思想活动的方式。这种AI技术很适合与机器习得配套使用。</p>
<p>DeepMind的AlphaGo AI能够打败围棋冠军也跟这些神经网络有很大关系。但是AlphaGo必须进行训练才行，开发人员向AlphaGo提供了历史对弈中的大约3000万记录。让人工智能技术具备通过记忆学习的能力，就可以让它独自完成更复杂的任务。</p>
<p>DeepMind希望DNC可以推动计算行业实现更多突破。DeepMind已将其研究结果发表在<a class="a-tips-Article-QQ" href="http://tech.qq.com/science.htm" target="_blank">科学</a>刊物《自然》（Nature）上。（编译/林靖东）</p>
<p><strong><strong>精彩视频推荐</strong></strong>
</p>
<div class="rv-root-v2 rv-js-root">
<div class="rv-middle">
<div class="rv-player" id="rv-player">
<div class="rv-player-adjust-img">
<div class="mbArticleSharePic">
<p class="mbArticleShareBtn"><span>转播到腾讯微博</span></p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAJAQMAAAAB5D5xAAAABGdBTUEAALGPC/xhBQAAAAFzUkdCAK7OHOkAAAADUExURQAAAKd6PdoAAAABdFJOUwBA5thmAAAAC0lEQVQI12NgwAkAABsAAVJE5KkAAAAASUVORK5CYIIvKiAgfHhHdjAwfDUwZDc5YmEzMGM3MDcxY2I5OTIyMTk4MzYyZGRlZmNlICov" /></div>
<div class="bd" accesskey="3" tabindex="-1">
<div id="Cnt-Main-Article-QQ" bosszone="content">
<div class="mbArticleSharePic">
<p class="mbArticleShareBtn"><span>转播到腾讯微博</span></p><img alt="DeepMind新电脑已可利用记忆自学 人工智能迈上新台阶" src="http://img1.gtimg.com/tech/pics/hv1/168/240/2140/139214868.jpg" />
</div>
<p>TNW中文站 10月14日报道</p>
<p><span onmouseover='ShowInfo(this,"GOOG.OQ","200","-1",event);'><a class="a-tips-Article-QQ" href="http://stockhtm.finance.qq.com/astock/ggcx/GOOG.OQ.htm" target="_blank">谷歌</a></span>(<a href="http://t.qq.com/googlechina#pref=qqcom.keyword" rel="googlechina" reltitle="谷歌" target="_blank">微博</a>) 在2014年收购的人工智能公司DeepMind开发出一款能够用自己的记忆学习新知识并利用这些知识来回答问题的计算机。</p>
<p>这款产品具有极其重要的意义，因为这意味着未来的人工智能技术可能不需要人类来教它就能回答人类提出的问题。</p>
<p>DeepMind表示，这款名为DNC（可微神经计算机）的AI模型可以接受家谱和伦敦地铁网络地图这样的信息，还可以回答与那些数据结构中的不同项目之间的关系有关的复杂问题。</p>
<p>例如，它可以回答“从邦德街开始，沿着中央线坐一站，环线坐四站，然后转朱比利线坐两站，你会到达哪个站？”这样的问题。</p>
<p>DeepMind称，DNC还可以帮你规划从沼泽门到皮卡迪利广场的最佳路线。</p>
<p>同样，它还可以理解和回答某个大家族中的成员之间的关系这样的复杂问题，比如“张三的大舅是谁？”。</p>
<p>DNC建立在神经网络的概念之上，神经网络可以模拟人类思想活动的方式。这种AI技术很适合与机器习得配套使用。</p>
<p>DeepMind的AlphaGo AI能够打败围棋冠军也跟这些神经网络有很大关系。但是AlphaGo必须进行训练才行，开发人员向AlphaGo提供了历史对弈中的大约3000万记录。让人工智能技术具备通过记忆学习的能力，就可以让它独自完成更复杂的任务。</p>
<p>DeepMind希望DNC可以推动计算行业实现更多突破。DeepMind已将其研究结果发表在<a class="a-tips-Article-QQ" href="http://tech.qq.com/science.htm" target="_blank">科学</a>刊物《自然》（Nature）上。（编译/林靖东）</p>
<p><strong><strong>精彩视频推荐</strong></strong>
</p>
<div class="rv-root-v2 rv-js-root">
<div class="rv-middle">
<div class="rv-player" id="rv-player">
<div class="rv-player-adjust-img">
<div class="mbArticleSharePic">
<p class="mbArticleShareBtn"><span>转播到腾讯微博</span></p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAJAQMAAAAB5D5xAAAABGdBTUEAALGPC/xhBQAAAAFzUkdCAK7OHOkAAAADUExURQAAAKd6PdoAAAABdFJOUwBA5thmAAAAC0lEQVQI12NgwAkAABsAAVJE5KkAAAAASUVORK5CYIIvKiAgfHhHdjAwfDUwZDc5YmEzMGM3MDcxY2I5OTIyMTk4MzYyZGRlZmNlICov" /></div>
</div>
</div>
</div>
</div>
<p><strong>【美国The Next Web作品的中文相关权益归腾讯公司独家所有。未经授权，不得转载、摘编等。】</strong></p>
</div>
<p><strong>【美国The Next Web作品的中文相关权益归腾讯公司独家所有。未经授权，不得转载、摘编等。】</strong></p>
</div>
</div>

24
test/test-pages/rtl-1/expected.html
Unescape Escape View File

@ -1,13 +1,15 @@
<div id="readability-page-1" class="page">
<article>
<p>
Lorem ipsum dolor sit amet.
</p>
<p>
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</p>
<p>
Tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</p>
</article>
<div>
<article>
<p>
Lorem ipsum dolor sit amet.
</p>
<p>
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</p>
<p>
Tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</p>
</article>
</div>
</div>

24
test/test-pages/rtl-2/expected.html
Unescape Escape View File

@ -1,13 +1,15 @@
<div id="readability-page-1" class="page">
<article>
<p>
Lorem ipsum dolor sit amet.
</p>
<p>
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</p>
<p>
Tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</p>
</article>
<div>
<article>
<p>
Lorem ipsum dolor sit amet.
</p>
<p>
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</p>
<p>
Tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</p>
</article>
</div>
</div>

24
test/test-pages/rtl-3/expected.html
Unescape Escape View File

@ -1,13 +1,15 @@
<div id="readability-page-1" class="page">
<article>
<p>
Lorem ipsum dolor sit amet.
</p>
<p>
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</p>
<p>
Tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</p>
</article>
<div dir="rtl">
<article>
<p>
Lorem ipsum dolor sit amet.
</p>
<p>
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</p>
<p>
Tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</p>
</article>
</div>
</div>

2
test/test-pages/rtl-4/expected-metadata.json
Unescape Escape View File

@ -2,6 +2,6 @@
"title": "RTL Test",
"byline": null,
"excerpt": "Lorem ipsum dolor sit amet.",
"dir": "rtl",
"dir": null,
"readerable": true
}

24
test/test-pages/rtl-4/expected.html
Unescape Escape View File

@ -1,13 +1,15 @@
<div id="readability-page-1" class="page">
<article dir="rtl">
<p>
Lorem ipsum dolor sit amet.
</p>
<p>
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</p>
<p>
Tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</p>
</article>
<div>
<article dir="rtl">
<p>
Lorem ipsum dolor sit amet.
</p>
<p>
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</p>
<p>
Tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</p>
</article>
</div>
</div>

1
test/test-pages/webmd-1/expected.html
Unescape Escape View File

@ -1,6 +1,5 @@
<div id="readability-page-1" class="page">
<div id="textArea" class="copyNormal">
<h3></h3>
<p>Feb. 23, 2015 -- Life-threatening peanut allergies have mysteriously been on the rise in the past decade, with little hope for a cure.</p>
<p xmlns:xalan="http://xml.apache.org/xalan">But a groundbreaking new study may offer a way to stem that rise, while another may offer some hope for those who are already allergic.</p>
<p>Parents have been told for years to avoid giving foods containing peanuts to babies for fear of triggering an allergy. Now research shows the opposite is true: Feeding babies snacks made with peanuts before their first birthday appears to prevent that from happening.</p>

1
test/test-pages/webmd-2/expected.html
Unescape Escape View File

@ -1,6 +1,5 @@
<div id="readability-page-1" class="page">
<div id="textArea" class="copyNormal">
<h3></h3>
<p>April 17, 2015 -- Imagine being sick in the hospital with a <a href="http://www.webmd.com/a-to-z-guides/bacterial-and-viral-infections" onclick="return sl(this,'','embd-lnk');" class="">bacterial infection</a> and doctors can't stop it from spreading. This so-called "superbug" scenario is not science fiction. It's an urgent, worldwide worry that is prompting swift action.</p>
<p xmlns:xalan="http://xml.apache.org/xalan">Every year, about 2 million people get sick from a superbug, according to the CDC. About 23,000 die. Earlier this year, an outbreak of CRE (carbapenem-resistant enterobacteriaceae) linked to contaminated medical tools sickened 11 people at two Los-Angeles area hospitals. Two people died, and more than 200 others may have been exposed.</p>
<p>The White House recently released a <a onclick="return sl(this,'','embd-lnk');" href="http://www.webmd.com/click?url=https://www.whitehouse.gov/sites/default/files/docs/national_action_plan_for_combating_antibotic-resistant_bacteria.pdf">comprehensive plan</a> outlining steps to combat drug-resistant bacteria. The plan identifies three "urgent" and several "serious" threats. We asked infectious disease experts to explain what some of them are and when to worry.</p>

98
test/test-pages/yahoo-1/expected.html
Unescape Escape View File

@ -1,51 +1,53 @@
<div id="readability-page-1" class="page">
<div id="Col1-0-ContentCanvas" class="content-canvas Bgc(#fff) Pos(r) P(20px)--sm Pt(17px)--sm" data-reactid="407">
<article data-uuid="80b35014-fba3-377e-adc5-47fb44f61fa7" data-type="story" data-reactid="408">
<figure class="canvas-image Mx(a) canvas-atom Mt(0px) Mt(20px)--sm Mb(24px) Mb(22px)--sm" data-type="image" data-reactid="409">
<div class="Maw(100%) Pos(r) H(0)" data-reactid="410"><img alt="The PlayStation VR" class="Trsdu(.42s) StretchedBox W(100%) H(100%) ie-7_H(a)" src="http://l1.yimg.com/ny/api/res/1.2/589noY9BZNdmsUUQf6L1AQ--/YXBwaWQ9aGlnaGxhbmRlcjtzbT0xO3c9NzQ0O2g9NjY5/http://media.zenfs.com/en/homerun/feed_manager_auto_publish_494/4406ef57dcb40376c513903b03bef048" data-reactid="411" /></div>
<div class="Ov(h) Pos(r) Mah(80px)" data-reactid="413">
<figcaption class="C(#787d82) Fz(13px) Py(5px) Lh(1.5)" title="Sony’s PlayStation VR." data-reactid="414">
<p class="figure-caption" data-reactid="415">Sony’s PlayStation VR.</p>
</figcaption>
<div id="Col1-0-ContentCanvas-Proxy" data-reactid="406">
<div id="Col1-0-ContentCanvas" class="content-canvas Bgc(#fff) Pos(r) P(20px)--sm Pt(17px)--sm" data-reactid="407">
<article data-uuid="80b35014-fba3-377e-adc5-47fb44f61fa7" data-type="story" data-reactid="408">
<figure class="canvas-image Mx(a) canvas-atom Mt(0px) Mt(20px)--sm Mb(24px) Mb(22px)--sm" data-type="image" data-reactid="409">
<div class="Maw(100%) Pos(r) H(0)" data-reactid="410"><img alt="The PlayStation VR" class="Trsdu(.42s) StretchedBox W(100%) H(100%) ie-7_H(a)" src="http://l1.yimg.com/ny/api/res/1.2/589noY9BZNdmsUUQf6L1AQ--/YXBwaWQ9aGlnaGxhbmRlcjtzbT0xO3c9NzQ0O2g9NjY5/http://media.zenfs.com/en/homerun/feed_manager_auto_publish_494/4406ef57dcb40376c513903b03bef048" data-reactid="411" /></div>
<div class="Ov(h) Pos(r) Mah(80px)" data-reactid="413">
<figcaption class="C(#787d82) Fz(13px) Py(5px) Lh(1.5)" title="Sony’s PlayStation VR." data-reactid="414">
<p class="figure-caption" data-reactid="415">Sony’s PlayStation VR.</p>
</figcaption>
</div>
</figure>
<div class="canvas-body C(#26282a) Wow(bw) Cl(start) Mb(20px) Fz(15px) Lh(1.6) Ff($ff-secondary)" data-reactid="418">
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="419">Virtual reality has officially reached the consoles. And it’s pretty good! <a href="http://finance.yahoo.com/news/review-playstation-vr-is-comfortable-and-affordable-but-lacks-must-have-games-165053851.html">Sony’s PlayStation VR</a> is extremely comfortable and reasonably priced, and while it’s lacking killer apps, it’s loaded with lots of interesting ones.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="420">But which ones should you buy? I’ve played just about every launch game, and while some are worth your time, others you might want to skip. To help you decide what’s what, I’ve put together this list of the eight PSVR games worth considering.</p>
<h3 class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="421"><a href="https://www.playstation.com/en-us/games/rez-infinite-ps4/" rel="nofollow noopener noreferrer" target="_blank">“Rez Infinite” ($30)</a></h3>
<p class="iframe-wrapper Pos(r) My(20px) canvas-atom Mt(14px)--sm Mb(0)--sm" data-reactid="422"><iframe class="canvas-video-iframe Bdw(0) StretchedBox W(100%) H(100%)" data-type="videoIframe" src="https://www.youtube.com/embed/YlDxEOwj5j8" data-reactid="423"></iframe></p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="424">Beloved cult hit “Rez” gets the VR treatment to help launch the PSVR, and the results are terrific. It includes a fully remastered take on the original “Rez” – you zoom through a Matrix-like computer system, shooting down enemies to the steady beat of thumping electronica – but the VR setting makes it incredibly immersive. It gets better the more you play it, too; unlock the amazing Area X mode and you’ll find yourself flying, shooting and bobbing your head to some of the trippiest visuals yet seen in VR.</p>
<h3 class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="425"><a href="https://www.playstation.com/en-us/games/thumper-ps4/" rel="nofollow noopener noreferrer" target="_blank">“Thumper” ($20)</a></h3>
<p class="iframe-wrapper Pos(r) My(20px) canvas-atom Mt(14px)--sm Mb(0)--sm" data-reactid="426"><iframe class="canvas-video-iframe Bdw(0) StretchedBox W(100%) H(100%)" data-type="videoIframe" src="https://www.youtube.com/embed/gtPGX8i1Eaw" data-reactid="427"></iframe></p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="428">What would happen if Tron, the board game Simon, a Clown beetle, Cthulhu and a noise band met in VR? Chaos, for sure, and also “Thumper.” Called a “violent rhythm game” by its creators, “Thumper” is, well, a violent rhythm game that’s also a gorgeous, unsettling and totally captivating assault on the senses. With simple controls and a straightforward premise – click the X button and the analog stick in time with the music as you barrel down a neon highway — it’s one of the rare games that works equally well both in and out of VR. But since you have PSVR, play it there. It’s marvelous.</p>
<h3 class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="429"><a href="https://www.playstation.com/en-us/games/until-dawn-rush-of-blood-ps4/" rel="nofollow noopener noreferrer" target="_blank">“Until Dawn: Rush of Blood” ($20)</a></h3>
<p class="iframe-wrapper Pos(r) My(20px) canvas-atom Mt(14px)--sm Mb(0)--sm" data-reactid="430"><iframe class="canvas-video-iframe Bdw(0) StretchedBox W(100%) H(100%)" data-type="videoIframe" src="https://www.youtube.com/embed/EL3svUfC8Ds" data-reactid="431"></iframe></p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="432">Cheeky horror game “Until Dawn” was a breakout hit for the PS4 last year, channeling the classic “dumb teens in the woods” horror trope into an effective interactive drama. Well, forget all that if you fire up “Rush of Blood,” because this one sticks you front and center on a rollercoaster ride from Hell. Literally. You ride through a dimly-lit carnival of terror, dual-wielding pistols as you take down targets, hideous pig monsters and, naturally, maniac clowns. Be warned: If the bad guys don’t get you, the jump scares will.</p>
<h3 class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="433"><a href="https://www.playstation.com/en-us/games/headmaster-ps4/" rel="nofollow noopener noreferrer" target="_blank">“Headmaster” ($20)</a></h3>
<p class="iframe-wrapper Pos(r) My(20px) canvas-atom Mt(14px)--sm Mb(0)--sm" data-reactid="434"><iframe class="canvas-video-iframe Bdw(0) StretchedBox W(100%) H(100%)" data-type="videoIframe" src="https://www.youtube.com/embed/a7CSMKw1E7g" data-reactid="435"></iframe></p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="436">Soccer meets “Portal” in the weird (and weirdly fun) “Headmaster,” a game about heading soccer balls into nets, targets and a variety of other things while stuck in some diabolical training facility. While at first it seems a little basic, increasingly challenging shots and a consistently entertaining narrative keep it from running off the pitch. Funny, ridiculous and as easy as literally moving your head back and forth, it’s a pleasant PSVR surprise.</p>
<h3 class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="437"><a href="https://www.playstation.com/en-us/games/rigs-mechanized-combat-league-ps4/" rel="nofollow noopener noreferrer" target="_blank">“RIGS: Mechanized Combat League” ($50)</a></h3>
<p class="iframe-wrapper Pos(r) My(20px) canvas-atom Mt(14px)--sm Mb(0)--sm" data-reactid="438"><iframe class="canvas-video-iframe Bdw(0) StretchedBox W(100%) H(100%)" data-type="videoIframe" src="https://www.youtube.com/embed/Rnqlf9EQ2zA" data-reactid="439"></iframe></p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="440">Giant mechs + sports? That’s the gist of this robotic blast-a-thon, which pits two teams of three against one another in gorgeous, explosive and downright fun VR combat. At its best, “RIGS” marries the thrill of fast-paced competitive shooters with the insanity of piloting a giant mech in VR. It can, however, be one of the barfier PSVR games. So pack your Dramamine, you’re going to have to ease yourself into this one.</p>
<h3 class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="441"><a href="https://www.playstation.com/en-us/games/batman-arkham-vr-ps4/" rel="nofollow noopener noreferrer" target="_blank">“Batman Arkham VR” ($20)</a></h3>
<p class="iframe-wrapper Pos(r) My(20px) canvas-atom Mt(14px)--sm Mb(0)--sm" data-reactid="442"><iframe class="canvas-video-iframe Bdw(0) StretchedBox W(100%) H(100%)" data-type="videoIframe" src="https://www.youtube.com/embed/eS4g0py16N8" data-reactid="443"></iframe></p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="444">“I’m Batman,” you will say. And you’ll actually be right this time, because you are Batman in this detective yarn, and you know this because you actually grab the famous cowl and mask, stick it on your head, and stare into the mirrored reflection of Rocksteady Games’ impressive Dark Knight character model. It lacks the action of its fellow “Arkham” games and runs disappointingly short, but it’s a high-quality experience that really shows off how powerfully immersive VR can be.</p>
<h3 class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="445"><a href="https://www.playstation.com/en-us/games/job-simulator-the-2050-archives-ps4/" rel="nofollow noopener noreferrer" target="_blank">“Job Simulator” ($30)</a></h3>
<p class="iframe-wrapper Pos(r) My(20px) canvas-atom Mt(14px)--sm Mb(0)--sm" data-reactid="446"><iframe class="canvas-video-iframe Bdw(0) StretchedBox W(100%) H(100%)" data-type="videoIframe" src="https://www.youtube.com/embed/3-iMlQIGH8Y" data-reactid="447"></iframe></p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="448">There are a number of good VR ports in the PSVR launch lineup, but the HTC Vive launch game “Job Simulator” might be the best. Your task? Lots of tasks, actually, from cooking food to fixing cars to working in an office, all for robots, because did I mention you were in the future? Infinitely charming and surprisingly challenging, it’s a great showpiece for VR.</p>
<h3 class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="449"><a href="https://www.playstation.com/en-us/games/eve-valkyrie-ps4/" rel="nofollow noopener noreferrer" target="_blank">“Eve Valkyrie” ($60)</a></h3>
<p class="iframe-wrapper Pos(r) My(20px) canvas-atom Mt(14px)--sm Mb(0)--sm" data-reactid="450"><iframe class="canvas-video-iframe Bdw(0) StretchedBox W(100%) H(100%)" data-type="videoIframe" src="https://www.youtube.com/embed/0KFHw12CTbo" data-reactid="451"></iframe></p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="452">Already a hit on the Oculus Rift, this space dogfighting game was one of the first to really show off how VR can turn a traditional game experience into something special. It’s pricey and not quite as hi-res as the Rift version, but “Eve Valkyrie” does an admirable job filling the void left since “Battlestar Galactica” ended. Too bad there aren’t any Cylons in it (or are there?)</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="453"><em><strong>More games news:</strong></em></p>
<ul class="canvas-list Pstart(40px) Mt(1.5em) Mb(1.5em) List(d)" data-type="list" data-reactid="454">
<li data-reactid="455"><a href="https://www.yahoo.com/tech/skylanders-imaginators-will-let-you-create-and-3d-print-your-own-action-figure-143838550.html">‘Skylanders Imaginators’ will let you create and 3D print your own action figures</a></li>
<li data-reactid="456"><a href="https://www.yahoo.com/tech/review-high-flying-nba-2k17-has-a-career-year-184135248.html">Review: High-flying ‘NBA 2K17’ has a career year</a></li>
<li data-reactid="457"><a href="https://www.yahoo.com/tech/review-race-at-your-own-speed-in-big-beautiful-forza-horizon-3-195337170.html">Review: Race at your own speed in big, beautiful ‘Forza Horizon 3’</a></li>
<li data-reactid="458"><a href="https://www.yahoo.com/tech/sonys-playstation-4-pro-shows-promise-potential-161304037.html">Sony’s PlayStation 4 Pro shows promise, potential and plenty of pretty lighting</a></li>
<li data-reactid="459"><a href="https://www.yahoo.com/tech/review-madden-nfl-17-runs-000000394.html">Review: ‘Madden NFL 17’ runs hard, plays it safe</a></li>
</ul>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="460"><i>Ben Silverman is on Twitter at</i>
<a href="https://twitter.com/ben_silverman" target="_blank" rel="nofollow noopener noreferrer"> <i>ben_silverman</i></a><i>.</i></p>
</div>
</figure>
<div class="canvas-body C(#26282a) Wow(bw) Cl(start) Mb(20px) Fz(15px) Lh(1.6) Ff($ff-secondary)" data-reactid="418">
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="419">Virtual reality has officially reached the consoles. And it’s pretty good! <a href="http://finance.yahoo.com/news/review-playstation-vr-is-comfortable-and-affordable-but-lacks-must-have-games-165053851.html">Sony’s PlayStation VR</a> is extremely comfortable and reasonably priced, and while it’s lacking killer apps, it’s loaded with lots of interesting ones.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="420">But which ones should you buy? I’ve played just about every launch game, and while some are worth your time, others you might want to skip. To help you decide what’s what, I’ve put together this list of the eight PSVR games worth considering.</p>
<h3 class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="421"><a href="https://www.playstation.com/en-us/games/rez-infinite-ps4/" rel="nofollow noopener noreferrer" target="_blank">“Rez Infinite” ($30)</a></h3>
<p class="iframe-wrapper Pos(r) My(20px) canvas-atom Mt(14px)--sm Mb(0)--sm" data-reactid="422"><iframe class="canvas-video-iframe Bdw(0) StretchedBox W(100%) H(100%)" data-type="videoIframe" src="https://www.youtube.com/embed/YlDxEOwj5j8" data-reactid="423"></iframe></p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="424">Beloved cult hit “Rez” gets the VR treatment to help launch the PSVR, and the results are terrific. It includes a fully remastered take on the original “Rez” – you zoom through a Matrix-like computer system, shooting down enemies to the steady beat of thumping electronica – but the VR setting makes it incredibly immersive. It gets better the more you play it, too; unlock the amazing Area X mode and you’ll find yourself flying, shooting and bobbing your head to some of the trippiest visuals yet seen in VR.</p>
<h3 class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="425"><a href="https://www.playstation.com/en-us/games/thumper-ps4/" rel="nofollow noopener noreferrer" target="_blank">“Thumper” ($20)</a></h3>
<p class="iframe-wrapper Pos(r) My(20px) canvas-atom Mt(14px)--sm Mb(0)--sm" data-reactid="426"><iframe class="canvas-video-iframe Bdw(0) StretchedBox W(100%) H(100%)" data-type="videoIframe" src="https://www.youtube.com/embed/gtPGX8i1Eaw" data-reactid="427"></iframe></p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="428">What would happen if Tron, the board game Simon, a Clown beetle, Cthulhu and a noise band met in VR? Chaos, for sure, and also “Thumper.” Called a “violent rhythm game” by its creators, “Thumper” is, well, a violent rhythm game that’s also a gorgeous, unsettling and totally captivating assault on the senses. With simple controls and a straightforward premise – click the X button and the analog stick in time with the music as you barrel down a neon highway — it’s one of the rare games that works equally well both in and out of VR. But since you have PSVR, play it there. It’s marvelous.</p>
<h3 class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="429"><a href="https://www.playstation.com/en-us/games/until-dawn-rush-of-blood-ps4/" rel="nofollow noopener noreferrer" target="_blank">“Until Dawn: Rush of Blood” ($20)</a></h3>
<p class="iframe-wrapper Pos(r) My(20px) canvas-atom Mt(14px)--sm Mb(0)--sm" data-reactid="430"><iframe class="canvas-video-iframe Bdw(0) StretchedBox W(100%) H(100%)" data-type="videoIframe" src="https://www.youtube.com/embed/EL3svUfC8Ds" data-reactid="431"></iframe></p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="432">Cheeky horror game “Until Dawn” was a breakout hit for the PS4 last year, channeling the classic “dumb teens in the woods” horror trope into an effective interactive drama. Well, forget all that if you fire up “Rush of Blood,” because this one sticks you front and center on a rollercoaster ride from Hell. Literally. You ride through a dimly-lit carnival of terror, dual-wielding pistols as you take down targets, hideous pig monsters and, naturally, maniac clowns. Be warned: If the bad guys don’t get you, the jump scares will.</p>
<h3 class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="433"><a href="https://www.playstation.com/en-us/games/headmaster-ps4/" rel="nofollow noopener noreferrer" target="_blank">“Headmaster” ($20)</a></h3>
<p class="iframe-wrapper Pos(r) My(20px) canvas-atom Mt(14px)--sm Mb(0)--sm" data-reactid="434"><iframe class="canvas-video-iframe Bdw(0) StretchedBox W(100%) H(100%)" data-type="videoIframe" src="https://www.youtube.com/embed/a7CSMKw1E7g" data-reactid="435"></iframe></p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="436">Soccer meets “Portal” in the weird (and weirdly fun) “Headmaster,” a game about heading soccer balls into nets, targets and a variety of other things while stuck in some diabolical training facility. While at first it seems a little basic, increasingly challenging shots and a consistently entertaining narrative keep it from running off the pitch. Funny, ridiculous and as easy as literally moving your head back and forth, it’s a pleasant PSVR surprise.</p>
<h3 class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="437"><a href="https://www.playstation.com/en-us/games/rigs-mechanized-combat-league-ps4/" rel="nofollow noopener noreferrer" target="_blank">“RIGS: Mechanized Combat League” ($50)</a></h3>
<p class="iframe-wrapper Pos(r) My(20px) canvas-atom Mt(14px)--sm Mb(0)--sm" data-reactid="438"><iframe class="canvas-video-iframe Bdw(0) StretchedBox W(100%) H(100%)" data-type="videoIframe" src="https://www.youtube.com/embed/Rnqlf9EQ2zA" data-reactid="439"></iframe></p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="440">Giant mechs + sports? That’s the gist of this robotic blast-a-thon, which pits two teams of three against one another in gorgeous, explosive and downright fun VR combat. At its best, “RIGS” marries the thrill of fast-paced competitive shooters with the insanity of piloting a giant mech in VR. It can, however, be one of the barfier PSVR games. So pack your Dramamine, you’re going to have to ease yourself into this one.</p>
<h3 class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="441"><a href="https://www.playstation.com/en-us/games/batman-arkham-vr-ps4/" rel="nofollow noopener noreferrer" target="_blank">“Batman Arkham VR” ($20)</a></h3>
<p class="iframe-wrapper Pos(r) My(20px) canvas-atom Mt(14px)--sm Mb(0)--sm" data-reactid="442"><iframe class="canvas-video-iframe Bdw(0) StretchedBox W(100%) H(100%)" data-type="videoIframe" src="https://www.youtube.com/embed/eS4g0py16N8" data-reactid="443"></iframe></p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="444">“I’m Batman,” you will say. And you’ll actually be right this time, because you are Batman in this detective yarn, and you know this because you actually grab the famous cowl and mask, stick it on your head, and stare into the mirrored reflection of Rocksteady Games’ impressive Dark Knight character model. It lacks the action of its fellow “Arkham” games and runs disappointingly short, but it’s a high-quality experience that really shows off how powerfully immersive VR can be.</p>
<h3 class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="445"><a href="https://www.playstation.com/en-us/games/job-simulator-the-2050-archives-ps4/" rel="nofollow noopener noreferrer" target="_blank">“Job Simulator” ($30)</a></h3>
<p class="iframe-wrapper Pos(r) My(20px) canvas-atom Mt(14px)--sm Mb(0)--sm" data-reactid="446"><iframe class="canvas-video-iframe Bdw(0) StretchedBox W(100%) H(100%)" data-type="videoIframe" src="https://www.youtube.com/embed/3-iMlQIGH8Y" data-reactid="447"></iframe></p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="448">There are a number of good VR ports in the PSVR launch lineup, but the HTC Vive launch game “Job Simulator” might be the best. Your task? Lots of tasks, actually, from cooking food to fixing cars to working in an office, all for robots, because did I mention you were in the future? Infinitely charming and surprisingly challenging, it’s a great showpiece for VR.</p>
<h3 class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="449"><a href="https://www.playstation.com/en-us/games/eve-valkyrie-ps4/" rel="nofollow noopener noreferrer" target="_blank">“Eve Valkyrie” ($60)</a></h3>
<p class="iframe-wrapper Pos(r) My(20px) canvas-atom Mt(14px)--sm Mb(0)--sm" data-reactid="450"><iframe class="canvas-video-iframe Bdw(0) StretchedBox W(100%) H(100%)" data-type="videoIframe" src="https://www.youtube.com/embed/0KFHw12CTbo" data-reactid="451"></iframe></p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="452">Already a hit on the Oculus Rift, this space dogfighting game was one of the first to really show off how VR can turn a traditional game experience into something special. It’s pricey and not quite as hi-res as the Rift version, but “Eve Valkyrie” does an admirable job filling the void left since “Battlestar Galactica” ended. Too bad there aren’t any Cylons in it (or are there?)</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="453"><em><strong>More games news:</strong></em></p>
<ul class="canvas-list Pstart(40px) Mt(1.5em) Mb(1.5em) List(d)" data-type="list" data-reactid="454">
<li data-reactid="455"><a href="https://www.yahoo.com/tech/skylanders-imaginators-will-let-you-create-and-3d-print-your-own-action-figure-143838550.html">‘Skylanders Imaginators’ will let you create and 3D print your own action figures</a></li>
<li data-reactid="456"><a href="https://www.yahoo.com/tech/review-high-flying-nba-2k17-has-a-career-year-184135248.html">Review: High-flying ‘NBA 2K17’ has a career year</a></li>
<li data-reactid="457"><a href="https://www.yahoo.com/tech/review-race-at-your-own-speed-in-big-beautiful-forza-horizon-3-195337170.html">Review: Race at your own speed in big, beautiful ‘Forza Horizon 3’</a></li>
<li data-reactid="458"><a href="https://www.yahoo.com/tech/sonys-playstation-4-pro-shows-promise-potential-161304037.html">Sony’s PlayStation 4 Pro shows promise, potential and plenty of pretty lighting</a></li>
<li data-reactid="459"><a href="https://www.yahoo.com/tech/review-madden-nfl-17-runs-000000394.html">Review: ‘Madden NFL 17’ runs hard, plays it safe</a></li>
</ul>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text" data-reactid="460"><i>Ben Silverman is on Twitter at</i>
<a href="https://twitter.com/ben_silverman" target="_blank" rel="nofollow noopener noreferrer"> <i>ben_silverman</i></a><i>.</i></p>
</div>
</article><span class="canvas-bottom-anchor-80b35014-fba3-377e-adc5-47fb44f61fa7" aria-hidden="true" data-reactid="462"></span></div>
</article><span class="canvas-bottom-anchor-80b35014-fba3-377e-adc5-47fb44f61fa7" aria-hidden="true" data-reactid="462"></span></div>
</div>
</div>

58
test/test-pages/yahoo-2/expected.html
Unescape Escape View File

@ -1,31 +1,33 @@
<div id="readability-page-1" class="page">
<div id="tgt1-Col1-2-ContentCanvas" class="content-canvas Bgc(#fff) Pos(r) P(20px)--sm Pt(17px)--sm">
<article data-uuid="8dd27580-6b4e-3cfb-a389-5b1ab90bd0eb" data-type="story">
<div class="canvas-atom Mb(24px) Mb(22px)--sm">
<div class="Bdbc(#e8e8e8) Bdbs(s) Bdbw(1px)">
<div class="Pos(r) Lh(1.4) Fz(13px) Pb(10px) Pt(6px)">
<p class="Fw(500) Fz(19px) Mb(8px)"><span>1 / 5</span></p>
<div class="Pos(r) Ovy(h)">
<div>
<p class="slideshow-description">In this photo dated Tuesday, Nov, 29, 2016 the Soyuz-FG rocket booster with the Progress MS-04 cargo ship is installed on a launch pad in Baikonur, Kazakhstan. The unmanned Russian cargo space ship Progress MS-04 broke up in the atmosphere over Siberia on Thursday Dec. 1, 2016, just minutes after the launch en route to the International Space Station due to an unspecified malfunction, the Russian space agency said.(Oleg Urusov/ Roscosmos Space Agency Press Service photo via AP)</p>
</div>
</div></div>
<div id="tgt1-Col1-2-ContentCanvas-Proxy">
<div id="tgt1-Col1-2-ContentCanvas" class="content-canvas Bgc(#fff) Pos(r) P(20px)--sm Pt(17px)--sm">
<article data-uuid="8dd27580-6b4e-3cfb-a389-5b1ab90bd0eb" data-type="story">
<div class="canvas-atom Mb(24px) Mb(22px)--sm">
<div class="Bdbc(#e8e8e8) Bdbs(s) Bdbw(1px)">
<div class="Pos(r) Lh(1.4) Fz(13px) Pb(10px) Pt(6px)">
<p class="Fw(500) Fz(19px) Mb(8px)"><span>1 / 5</span></p>
<div class="Pos(r) Ovy(h)">
<div>
<p class="slideshow-description">In this photo dated Tuesday, Nov, 29, 2016 the Soyuz-FG rocket booster with the Progress MS-04 cargo ship is installed on a launch pad in Baikonur, Kazakhstan. The unmanned Russian cargo space ship Progress MS-04 broke up in the atmosphere over Siberia on Thursday Dec. 1, 2016, just minutes after the launch en route to the International Space Station due to an unspecified malfunction, the Russian space agency said.(Oleg Urusov/ Roscosmos Space Agency Press Service photo via AP)</p>
</div>
</div></div>
</div>
</div>
</div>
<div class="canvas-body C(#26282a) Wow(bw) Cl(start) Mb(20px) Fz(15px) Lh(1.6) Ff($ff-secondary)">
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">MOSCOW (AP) — An unmanned Russian cargo spaceship heading to the International Space Station broke up in the atmosphere over Siberia on Thursday due to an unspecified malfunction, the Russian space agency said.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">The Progress MS-04 cargo craft broke up at an altitude of 190 kilometers (118 miles) over the remote Russian Tuva region in Siberia that borders Mongolia, Roscosmos said in a statement. It said most of spaceship's debris burnt up as it entered the atmosphere but some fell to Earth over what it called an uninhabited area.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">Local people reported seeing a flash of light and hearing a loud thud west of the regional capital of Kyzyl, more than 3,600 kilometers (2,200 miles) east of Moscow, the Tuva government was quoted as saying late Thursday by the Interfax news agency.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">The Progress cargo ship had lifted off as scheduled at 8:51 p.m. (1451 GMT) from Russia's space launch complex in Baikonur, Kazakhstan, to deliver 2.5 metric tons of fuel, water, food and other supplies. It was set to dock with the space station on Saturday.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">Roscosmos said the craft was operating normally before it stopped transmitting data 6 ½ minutes after the launch. The Russian space agency would not immediately describe the malfunction, saying its experts were looking into it.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">This is the third botched launch of a Russian spacecraft in two years. A Progress cargo ship plunged into the Pacific Ocean in May 2015, and a Proton-M rocket carrying an advanced satellite broke up in the atmosphere in May 2014.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">But both Roscosmos and NASA said the crash of the ship would have no impact on the operations of the orbiting space lab that is currently home to a six-member crew, including three cosmonauts from Russia, two NASA astronauts and one from the European Union.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">Orbital ATK, NASA's other shipper, successfully sent up supplies to the space station in October, and a Japanese cargo spaceship is scheduled to launch a full load in mid-December.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">NASA supplier SpaceX, meanwhile, has been grounded since a rocket explosion in September on the launch pad at Cape Canaveral, Florida. The company hopes to resume launches in December to deliver communication satellites.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">___</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">This version corrects the spelling of the region to Tuva, not Tyva.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">__</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">Aerospace Writer Marcia Dunn in Cape Canaveral, Florida, and Vladimir Isachenkov in Moscow contributed to this report.</p>
</div>
</article><span class="canvas-bottom-anchor-8dd27580-6b4e-3cfb-a389-5b1ab90bd0eb" aria-hidden="true"></span></div>
<div class="canvas-body C(#26282a) Wow(bw) Cl(start) Mb(20px) Fz(15px) Lh(1.6) Ff($ff-secondary)">
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">MOSCOW (AP) — An unmanned Russian cargo spaceship heading to the International Space Station broke up in the atmosphere over Siberia on Thursday due to an unspecified malfunction, the Russian space agency said.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">The Progress MS-04 cargo craft broke up at an altitude of 190 kilometers (118 miles) over the remote Russian Tuva region in Siberia that borders Mongolia, Roscosmos said in a statement. It said most of spaceship's debris burnt up as it entered the atmosphere but some fell to Earth over what it called an uninhabited area.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">Local people reported seeing a flash of light and hearing a loud thud west of the regional capital of Kyzyl, more than 3,600 kilometers (2,200 miles) east of Moscow, the Tuva government was quoted as saying late Thursday by the Interfax news agency.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">The Progress cargo ship had lifted off as scheduled at 8:51 p.m. (1451 GMT) from Russia's space launch complex in Baikonur, Kazakhstan, to deliver 2.5 metric tons of fuel, water, food and other supplies. It was set to dock with the space station on Saturday.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">Roscosmos said the craft was operating normally before it stopped transmitting data 6 ½ minutes after the launch. The Russian space agency would not immediately describe the malfunction, saying its experts were looking into it.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">This is the third botched launch of a Russian spacecraft in two years. A Progress cargo ship plunged into the Pacific Ocean in May 2015, and a Proton-M rocket carrying an advanced satellite broke up in the atmosphere in May 2014.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">But both Roscosmos and NASA said the crash of the ship would have no impact on the operations of the orbiting space lab that is currently home to a six-member crew, including three cosmonauts from Russia, two NASA astronauts and one from the European Union.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">Orbital ATK, NASA's other shipper, successfully sent up supplies to the space station in October, and a Japanese cargo spaceship is scheduled to launch a full load in mid-December.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">NASA supplier SpaceX, meanwhile, has been grounded since a rocket explosion in September on the launch pad at Cape Canaveral, Florida. The company hopes to resume launches in December to deliver communication satellites.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">___</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">This version corrects the spelling of the region to Tuva, not Tyva.</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">__</p>
<p class="canvas-text Mb(1.0em) Mb(0)--sm Mt(0.8em)--sm canvas-atom" data-type="text">Aerospace Writer Marcia Dunn in Cape Canaveral, Florida, and Vladimir Isachenkov in Moscow contributed to this report.</p>
</div>
</article><span class="canvas-bottom-anchor-8dd27580-6b4e-3cfb-a389-5b1ab90bd0eb" aria-hidden="true"></span></div>
</div>
</div>

78
test/test-pages/yahoo-3/expected.html
Unescape Escape View File

@ -1,42 +1,44 @@
<div id="readability-page-1" class="page">
<div id="Main" tabindex="0" class="Pos-r Z-3 Stack yog-card yog-content" role="main">
<section class="yom-mod yom-breaking-news level2" id="mediacontentbreakingnews" data-ylk="mid:mediacontentbreakingnews;mpos:1;t1:a3;t2:mod-bkn;sec:mod-bkn;">
<p class="hd"> <span>'GMA' Cookie Search:</span> </p>
</section>
<section id="mediacontentstory" class="yom-mod yom-card-main yom-article" data-uuid="4250eebf-bbb0-3c95-8fd0-3cb4d3daf93c" data-type="story" data-ylk="t1:a3;t2:ct-mod;sec:ct-mod;itc:0;rspns:nav;">
<div class="book clearfix">
<div class="body yom-art-content clearfix" itemscope="" itemtype="https://schema.org/Article">
<meta itemprop="datePublished" content="2015-03-11T19:46:14Z" />
<meta itemprop="headline" content="Veteran Wraps Baby in American Flag, Photo Sparks Controversy" />
<meta itemprop="alternativeHeadline" content="" />
<meta itemprop="image" content="https://media.zenfs.com/en-US/video/video.abcnewsplus.com/559ecdbafdb839129816b5c79a996975" />
<meta itemprop="description" content="A photographer and Navy veteran is fighting back after a photo she posted to Facebook started an online backlash. Vanessa Hicks said she had no idea her photo would be considered controversial. The photo, from a military family’s newborn photo shoot, showed a newborn infant wrapped in an American flag held by his father, who was in his military uniform. Hicks, a Navy veteran herself and the wife of an active-duty Navy member, said her intention was to honor the flag as well as her clients, who wanted to incorporate their military service in the photo shoot." />
<p>A photographer and Navy veteran is fighting back after a photo she posted to <a id="ramplink_Facebook_" href="http://abcnews.go.com/topics/business/companies/facebook-inc.htm" target="_blank" data-rapid_p="13">Facebook</a> started an online backlash.</p>
<p>Vanessa Hicks said she had no idea her photo would be considered controversial. The photo, from a military family’s newborn photo shoot, showed a newborn infant wrapped in an American flag held by his father, who was in his military uniform.</p>
<p>Hicks, a Navy veteran herself and the wife of an active-duty Navy member, said her intention was to honor the flag as well as her clients, who wanted to incorporate their military service in the photo shoot.</p>
<p><a href="http://abcnews.go.com/WNT/video/pizza-patriots-making-special-super-bowl-delivery-troops-28633975" target="_blank" data-rapid_p="14">Pizza Man Making Special Delivery Pizza Delivery to Afghanistan During Super Bowl</a></p>
<p><a href="http://abcnews.go.com/Health/superbug-scope-maker-altered-design-make-cleaning-easier/story?id=29417816" target="_blank" data-rapid_p="15">Redesigned Scopes Fail to Stop 'Superbug Outbreaks</a></p>
<p><a href="http://abcnews.go.com/Travel/video/antarctica-penguin-post-office-job-attracts-record-number-29247380" target="_blank" data-rapid_p="16">Antarctica 'Penguin Post Office' Attracts Record Number of Applicants</a></p>
<p>“This is what he was fighting for, his son wrapped in an American flag,” Hicks told ABC News. However, when she posted the image on her page, she started to get comments accusing her of desecrating the flag.</p>
<p>On one Facebook page an unidentified poster put up her picture writing and wrote they found it was “disrespectful, rude, tacky, disgusting, and against the U.S. Flag Code.”</p>
<div class="yom-fig-frame"><span id="schemaorg"><div class="yom-figure yom-fig-middle"><figure class="cover get-lbdata-from-dom go-to-slideshow-lightbox" data-orig-index="2"> <a class="cover-anchor" name="cover-c9b69c1a26e19ae9fe744763dc31e9ac" id="cover-c9b69c1a26e19ae9fe744763dc31e9ac" data-rapid_p="17"></a><div class="cta-overlay"><span class="clearfix title cta-text medium"></span>
<p class="cta-text large">View photo</p><span class="icon-slideshow icon-white-slideshow">.</span></div><img alt="Vanessa Hicks" class="editorial lzbg" data-preembed="image" src="https://s3.yimg.com/bt/api/res/1.2/GNtA09EDJWzWfpBzGYJS0Q--/YXBwaWQ9eW5ld3NfbGVnbztxPTg1O3c9NjMw/http://media.zenfs.com/en_us/gma/us.abcnews.gma.com/HT_flag_baby_jtm_150311_16x9_992.jpg" title="Vanessa Hicks" width="630" /></figure>
<p class="legend">Vanessa Hicks</p>
<div class="Col2 yog-cp Pos-a Bxz-bb">
<div id="Main" tabindex="0" class="Pos-r Z-3 Stack yog-card yog-content" role="main">
<section class="yom-mod yom-breaking-news level2" id="mediacontentbreakingnews" data-ylk="mid:mediacontentbreakingnews;mpos:1;t1:a3;t2:mod-bkn;sec:mod-bkn;">
<p class="hd"> <span>'GMA' Cookie Search:</span> </p>
</section>
<section id="mediacontentstory" class="yom-mod yom-card-main yom-article" data-uuid="4250eebf-bbb0-3c95-8fd0-3cb4d3daf93c" data-type="story" data-ylk="t1:a3;t2:ct-mod;sec:ct-mod;itc:0;rspns:nav;">
<div class="book clearfix">
<div class="body yom-art-content clearfix" itemscope="" itemtype="https://schema.org/Article">
<meta itemprop="datePublished" content="2015-03-11T19:46:14Z" />
<meta itemprop="headline" content="Veteran Wraps Baby in American Flag, Photo Sparks Controversy" />
<meta itemprop="alternativeHeadline" content="" />
<meta itemprop="image" content="https://media.zenfs.com/en-US/video/video.abcnewsplus.com/559ecdbafdb839129816b5c79a996975" />
<meta itemprop="description" content="A photographer and Navy veteran is fighting back after a photo she posted to Facebook started an online backlash. Vanessa Hicks said she had no idea her photo would be considered controversial. The photo, from a military family’s newborn photo shoot, showed a newborn infant wrapped in an American flag held by his father, who was in his military uniform. Hicks, a Navy veteran herself and the wife of an active-duty Navy member, said her intention was to honor the flag as well as her clients, who wanted to incorporate their military service in the photo shoot." />
<p>A photographer and Navy veteran is fighting back after a photo she posted to <a id="ramplink_Facebook_" href="http://abcnews.go.com/topics/business/companies/facebook-inc.htm" target="_blank" data-rapid_p="13">Facebook</a> started an online backlash.</p>
<p>Vanessa Hicks said she had no idea her photo would be considered controversial. The photo, from a military family’s newborn photo shoot, showed a newborn infant wrapped in an American flag held by his father, who was in his military uniform.</p>
<p>Hicks, a Navy veteran herself and the wife of an active-duty Navy member, said her intention was to honor the flag as well as her clients, who wanted to incorporate their military service in the photo shoot.</p>
<p><a href="http://abcnews.go.com/WNT/video/pizza-patriots-making-special-super-bowl-delivery-troops-28633975" target="_blank" data-rapid_p="14">Pizza Man Making Special Delivery Pizza Delivery to Afghanistan During Super Bowl</a></p>
<p><a href="http://abcnews.go.com/Health/superbug-scope-maker-altered-design-make-cleaning-easier/story?id=29417816" target="_blank" data-rapid_p="15">Redesigned Scopes Fail to Stop 'Superbug Outbreaks</a></p>
<p><a href="http://abcnews.go.com/Travel/video/antarctica-penguin-post-office-job-attracts-record-number-29247380" target="_blank" data-rapid_p="16">Antarctica 'Penguin Post Office' Attracts Record Number of Applicants</a></p>
<p>“This is what he was fighting for, his son wrapped in an American flag,” Hicks told ABC News. However, when she posted the image on her page, she started to get comments accusing her of desecrating the flag.</p>
<p>On one Facebook page an unidentified poster put up her picture writing and wrote they found it was “disrespectful, rude, tacky, disgusting, and against the U.S. Flag Code.”</p>
<div class="yom-fig-frame"><span id="schemaorg"><div class="yom-figure yom-fig-middle"><figure class="cover get-lbdata-from-dom go-to-slideshow-lightbox" data-orig-index="2"> <a class="cover-anchor" name="cover-c9b69c1a26e19ae9fe744763dc31e9ac" id="cover-c9b69c1a26e19ae9fe744763dc31e9ac" data-rapid_p="17"></a><div class="cta-overlay"><span class="clearfix title cta-text medium"></span>
<p class="cta-text large">View photo</p><span class="icon-slideshow icon-white-slideshow">.</span></div><img alt="Vanessa Hicks" class="editorial lzbg" data-preembed="image" src="https://s3.yimg.com/bt/api/res/1.2/GNtA09EDJWzWfpBzGYJS0Q--/YXBwaWQ9eW5ld3NfbGVnbztxPTg1O3c9NjMw/http://media.zenfs.com/en_us/gma/us.abcnews.gma.com/HT_flag_baby_jtm_150311_16x9_992.jpg" title="Vanessa Hicks" width="630" /></figure>
<p class="legend">Vanessa Hicks</p>
</div>
</span>
</div>
</span>
</div>
<p style="display: inline;" class="readability-styled">The Federal Flag Code has guidelines for the proper treatment of the U.S. Flag but there are no rules for punishment related to violations. In the past, the </p><a href="http://abcnews.go.com/topics/news/us/supreme-court.htm" target="_blank" data-rapid_p="18">Supreme Court</a>
<p style="display: inline;" class="readability-styled"> has found that people are protected from punishment under the First Amendment for manipulating or even burning the flag. </p>
<p>Hicks said she was surprised when messages suddenly started to pop up on her Facebook page and even her own website criticizing her photos.</p>
<p>She said she stayed up until 4 a.m. recently to take down comments from her business and company page, even on shoots that had nothing to do with the flag.</p>
<p>“I know how low I felt during those first few hours,” said Hicks. “[I felt] am I not a good American or veteran or wife. It’s a train-wreck you can’t help but watch.”</p>
<p>As Hicks tried to stop the comments from taking over her pages, others started to take notice and her picture went viral on social media sites. After that, Hicks found that many people, both military and civilian, told her they did not find the picture offensive.</p>
<p>“I have seen first-hand what is desecration of the flag,” Hicks said of her time in the military. “At the end of the day I didn’t do anything that disrespected this flag.”</p>
<p>Hicks, whose husband is still on active duty in the Navy, said the flag is a symbol of U.S. freedoms including the First Amendment right to free speech.</p>
<p>“[My husband] wouldn’t die for a flag, he would die for the freedoms that this country offers,” she told ABC News.</p>
<p>After her story grabbed local headlines, Hicks has been inundated by requests for photos shoots, and she said she plans to give 15 percent of all profits related to these shoots to the USO.</p>
<p style="display: inline;" class="readability-styled">The Federal Flag Code has guidelines for the proper treatment of the U.S. Flag but there are no rules for punishment related to violations. In the past, the </p><a href="http://abcnews.go.com/topics/news/us/supreme-court.htm" target="_blank" data-rapid_p="18">Supreme Court</a>
<p style="display: inline;" class="readability-styled"> has found that people are protected from punishment under the First Amendment for manipulating or even burning the flag. </p>
<p>Hicks said she was surprised when messages suddenly started to pop up on her Facebook page and even her own website criticizing her photos.</p>
<p>She said she stayed up until 4 a.m. recently to take down comments from her business and company page, even on shoots that had nothing to do with the flag.</p>
<p>“I know how low I felt during those first few hours,” said Hicks. “[I felt] am I not a good American or veteran or wife. It’s a train-wreck you can’t help but watch.”</p>
<p>As Hicks tried to stop the comments from taking over her pages, others started to take notice and her picture went viral on social media sites. After that, Hicks found that many people, both military and civilian, told her they did not find the picture offensive.</p>
<p>“I have seen first-hand what is desecration of the flag,” Hicks said of her time in the military. “At the end of the day I didn’t do anything that disrespected this flag.”</p>
<p>Hicks, whose husband is still on active duty in the Navy, said the flag is a symbol of U.S. freedoms including the First Amendment right to free speech.</p>
<p>“[My husband] wouldn’t die for a flag, he would die for the freedoms that this country offers,” she told ABC News.</p>
<p>After her story grabbed local headlines, Hicks has been inundated by requests for photos shoots, and she said she plans to give 15 percent of all profits related to these shoots to the USO.</p>
</div>
</div>
</section>
</div>
</div>
</div>
</section>
</div>
</div>
Write Preview
Loading…
Cancel
Save