Online Manga Reader Reader with node-webkit

In this tutorial I want to tell you a way to use node-webkit to create a desktop application to read your favorite online manga. Usually a manga reader display the image one at a time per page. Now with the app we are going to make, we will display all images in a chapter at a time.

Requirements

These are the tools you will need to do this:

Prepare The Workspace

Before we delve into the main code, let’s prepare the place where we will build the program. Create a new directory and create these three files:

package.json

{
  "name": "onlinemangareader",
  "main": "index.html",
  "nodejs": true,
  "window": {
    "toolbar": false,
    "width": 800,
    "height": 600
  }
}

package.json is the file needed by node-webkit to recognize your application. The complete explanation of this file can be found in its wiki.

index.html

<!DOCTYPE html>
<html>
  <head>
    <title>Online Manga Reader Reader</title>
  </head>
  <body>
    <h1>Hello World!</h1>
    We are using node.js <script>document.write(process.version)</script>.
  </body>
</html>

This is the file that will be loaded at program startup (as we stated in the package.json above). This is just an ordinary HTML file.

deploy.bat

/path/to/node-webkit/nw.exe .

This is just a shortcut script to run our source code with node-webkit.

Let’s try running the app by double-clicking the deploy.bat script.

hello

Research The Target

Our target is one of the many online manga websites, GoodManga.Net. First, we need to understand how each manga images are displayed in this website so we can build a web scraper to obtain this information automatically.

Let’s open any chapter on that website, for example Ojojojo chapter 19.

ojojojo 1

If you travel to another page, you can see a pattern in the URL:

Page 1  http://www.goodmanga.net/ojojojo/chapter/19
Page 2  http://www.goodmanga.net/ojojojo/chapter/19/2
...
Page 10 http://www.goodmanga.net/ojojojo/chapter/19/10

Let’s check the location of the image in the HTML source code. Do “view source” in one the page, most browser use Ctrl+U as shortcut. To find the image, you can skim over it and guess, or search all <img> tags, or you copy the URL of the image and then search it in the page source.

Either way you do it, you will find the image around this part of HTML:

...
<div id="manga_viewer">
  <div id="mv_ad_side_1" style="left: 0px;">
    <iframe width="160" scrolling="no" height="600" frameborder="0" src="/ads/mv_side_1.html"></iframe>
  </div>
  <a href="http://www.goodmanga.net/ojojojo/chapter/20"><img src="http://r1.goodmanga.net/images/manga/ojojojo/19/10.jpg" data-width="800" width="800" alt="Ojojojo   19 Page 10" /></a>

Now we know how to navigate to other page (change the last chunk of URL) and where the image located in a page (div#manga_viewer > a > img). The only thing left is how to determine how many pages are there in a chapter.

From the browser page, you can immediately tell that there are 10 pages, but a web scraper doesn’t see like you do. So we have to guide it, defining its location in HTML source. By quickly searching “of 10″ in the page source, we find the text are contained in this part:

...
<div id="manga_nav_top">
  <select name="chapter_select" class="chapter_select">
    <option selected="selected">Ojojojo   Chapter 19 </option>
  </select>
  <span>
    <a href="http://www.goodmanga.net/ojojojo/chapter/19/8" class="previous_page">Previous</a>
    <select name="page_select" class="page_select">
      <option selected="selected">9 </option>
    </select>
    <span>of 10</span>

That page count location is div#manga_nav_top > span > span.

If you are not familiar with CSS selector, “div#manga_viewer > a > img” means a <div> element with id=”manga_viewer” has direct child element <a> that has direct child element <img>, choose the <img> element.

Draft The Code

We are going to use node.js code to do this, because basic JavaScript can’t open other URL via Ajax (same origin policy). We will need 2 node package:

Let’s install those packages. How to install node package for node-webkit you ask? Just like a regular node app. In your app directory, run:

npm install request
npm install cheerio

This will create a new directory node_modules there.

node_modules/
    cheerio/
    request/

With the required packages available, we can start coding. Open that index.html file and replace the content of <body> tag with this:

<body>
  <div id="result"></div>
  <div id="loading"></div>

  <script>
  var request = require('request');
  var cheerio = require('cheerio');

  var url = 'http://www.goodmanga.net/ojojojo/chapter/19';
  var result = document.getElementById('result');
  var loading = document.getElementById('loading');

  scrap_first_page(url);

  function scrap_first_page(url) {
    request(url, function (error, response, html) {
      var $, img, n;

      if (!error) {
        $ = cheerio.load(html);
        img = get_image($);
        n = get_number_of_page($);
        print_image(img, 1);
        scrap_other_page(url, 2, n);
      }
    });
  }

  function scrap_other_page(base_url, i, n) {
    var url;

    if (i < n) {
      url = base_url + "/" + i;
      request(url, function (error, response, html) {
        var $, img;

        if (!error) {
          $ = cheerio.load(html);
          img = get_image($);
          print_image(img, i);
          scrap_other_page(base_url, i+1, n);
        } else {
          loading.innerHTML = 'ERROR OCCURED! PLEASE RESET';
        }
      });
    } else {
      loading.innerHTML = 'Done';
    }
  }

  function get_image($) {
    var $img = $('#manga_viewer').find('img').first();
    return $img.attr('src');
  }

  function get_number_of_page($) {
    var $n = $('#manga_nav_top')
      .find('span').first()
      .find('span').first();
    var match = $n.html().match(/of (\d+)/);
    return match[1];
  }

  function print_image(src, i) {
    result.innerHTML += "<img src='" + src + "'><br>";
    loading.innerHTML = "Loading page " + (i+1);
  }

  </script>
</body>

Take some time to understand the code above, I assume you understand DOM. The basic idea is:

  1. Open first page
  2. Get N <- number of page
  3. Get IMG <- URL of image in this page
  4. Print IMG
  5. For 2 <= I <= N
    1. Open I-th page
    2. Get IMG <- URL of image in I-th page
    3. Print IMG

But instead of looping, I use recursion (see function scrap_other_page()). This is because many operations in node.js is asynchronous, just like the Request library we use.

Let’s try this code, run the deploy.bat and see the images get loaded.

try draft

Give Input Field

We have managed to view a chapter, but the URL is still hardcoded. We want the ability to provide the URL via some kind of input. With that said, let’s add the input element. Just after the opening <body> tag, add this code:

<p>
  <input id="source_url"/>
  <button id="scrap" type="button">Scrap!</button>
</p>

We also need to make some modification in the JavaScript code, so change this code:

  var url = 'http://www.goodmanga.net/ojojojo/chapter/19';
  var result = document.getElementById('result');
  var loading = document.getElementById('loading');

  scrap_first_page(url);

With this:

  var result = document.getElementById('result');
  var loading = document.getElementById('loading');
  var source = document.getElementById('source_url');
  var scrap = document.getElementById('scrap');

  scrap.addEventListener('click', function() {
    var url = source.value;
    loading.innerHTML = 'Loading...';
    result.innerHTML = '';
    scrap_first_page(url);
  });

We create two new elements: text input and a trigger button. Then, we attach a “click” event to the button. Let’s hope this works as expected.

Try It

Run deploy.bat again and when the app is up, put a manga chapter URL in the input field then click the button. If images start to show up, congratulations! You did it!

final try

After all images are loaded, you can put another chapter URL in the input field and click the button. This will reset the page empty and load that chapter images.

Afterthought

This tutorial shows you how you can make your own web scraper with node-webkit. You also learn how node.js code is mixed with regular front-end JavaScript code. You can apply the same technique here to other similar website and make your own reader.

There are many improvements you can do to this app. For example, you can style it up a bit, also you can implement a cancel button. After this, you can package the app as an executable for easier distribution if you want.

node-webkit: TodoMVC as a Desktop App

In this tutorial I will explain to you how to use node-webkit to create a web app into desktop application. The web app we use is TodoMVC.

Obtain node-webkit

node-webkit is a web runtime based on Chromium and node.js. In simpler term, node-webkit can convert HTML app into desktop app. It does this by integrating a mini-browser to run the HTML app.

First, we need to install node-webkit. Luckily, node-webkit is a portable program so we just need to download it. The download link can be seen in its github page under “Downloads” section.

download nodewebkit

Choose the suitable program based on your operating system. Download and extract it somewhere.

Obtain TodoMVC

TodoMVC is a project which offers the same Todo application implemented using MV* concepts in most of the popular JavaScript MV* frameworks. So TodoMVC is a todo app written in HTML (+ JS + CSS).

To download TodoMVC, click on the big red button on its website.

download todomvc

The zip file that you download contains many TodoMVC apps source code implemented in many different tools. Let’s select one of it, I choose the angularjs implementation. So extract the content of \tastejs-todomvc-*\architecture-examples\angularjs\ into a directory. We will be working in that directory.

Try TodoMVC (optional)

In case you are wondering what TodoMVC looks like, go to the directory where you extracted it and open the index.html file with your web browser.

open indexhtmltodomvc browser

Package TodoMVC

For an app to be able to be converted into desktop app, node-webkit needs a special file called package.json. package.json is just a text file containing a JSON-formatted data describing the app.

Create a file in your TodoMVC directory

/path/to/your/todomvc/package.json

{
  "name": "todomvc",
  "main": "index.html",
  "window": {
    "toolbar": false,
    "width": 800,
    "height": 600
  }
}

Even without knowing exactly the meaning behind above code, you can guess that it tells node-webkit to open index.html file with window size 800 x 600 px. Complete explanation about package.json.

The second step is to zip the source codes. Name it whatever you want, e.g. app.zip.

zip app

After that rename the extension into .nw, e.g. app.nw. This is the extension required by node-webkit, it’s just zip file renamed.

rename zip

Create Executable

The next step is basically to merge node-webkit executable with your app.nw file you created earlier. This step is simple but the command is different per operating system.

On Windows

copy /b /path/to/node-webkit/nw.exe+app.nw app.exe

On Linux

On Mac OS X

Now, the combined executable is not enough for distribution. You need to include several files from node-webkit directory with that executable to make it run. The files needed are different per operating system.

On Windows you need: nw.pak, icudt.dat, ffmpegsumo.dll (if you use media feature), libEGL.dll & libGLESv2.dll (for WebGL & GPU acceleration).

distribute dll

On Linux you need: nw.pak and icudtl.dat

On Mac OS X you need: icudtl.dat

So, create a new directory, put the combined executable there and copy the necessary files above to that directory.

If you want to know more about packaging and distributing app.

Try It

If you have been following this tutorial, you will have a directory like this (on Windows)

final dir

Let’s try running that app.exe.

run app

Congratulations, that collection of html files have become a desktop executable.

Afterthought

So what is the point of using node-webkit? For me, I have been creating many html apps and sometimes I wonder if only I could make some the apps into desktop app, it would be easier to use and distribute. With node-webkit, not only that but you can also use node.js in your app.

1.5 Year Later

It has been about 1.5 year since the last blog post I made. Apparently I have abandoned this blog for so long. Recently I got some ideas of what to write so maybe I will be active again blogging for next couple of months, no promises.

Just to recap what happened in the last idle time:

  • You know that I wrote CodeIgniter Sucks
    • Just to clarify, I think CodeIgniter 2 really sucks, and with Ellislab abandoning it, just make it worst.
    • In the above document you can see that most of the issues are fixed in CodeIgniter 3 but CodeIgniter 3 is not released yet (beta/development version means not released).
    • Even if CodeIgniter 3 is released, I still don’t recommend CodeIgniter if you want to use PHP framework.
  • After more than a year in the new job, it has been going great. Learn many stuffs.
  • Yesterday I logged in this blog and found that all new comments are spam, figures.
  • Updated my About page.
  • I have been thinking about moving this blog to another hosting, if I am really serious about activating this blog again.

JavaScript OTR Browser Performance

So for the past few weeks I have been developing a JavaScript OTR library. It’s still in alpha stage right now although it can communicate with other OTR-compliant chat client. When the code is ready, I might release it for public use.

That aside, now I want to show you a simple benchmark of how different browser perform with this library.

UPDATE: I added android browser benchmark.

#1 The fastest one: Google Chrome

#2 A little behind: Mozilla Firefox (Without firebug)

#3 Almost three times slower than Firefox: Opera

#4 A bit slower than Opera: Safari for Windows

#5 Surprisingly faster than IE9: Dolphin Browser HD (Android 4.0.4 Aoson M11 tablet)

#6 The slowest desktop browser, far behind: Internet Explorer 9 (duh)

#7 Out of curiosity I tested my phone: Dolphin Browser Mini (Android 2.3.7 Huawei Ideos x5 phone)

I have only tested it on the above browsers.

Note: I use JSBN for BigInteger computation but I use Leemon‘s powMod for modular exponentiation because it’s faster.