#Wikipedia Zero - I am all for it II

Wikipedia Zero is a tool. Its purpose is to enable more people reading Wikipedia. The problem it solves is that many people have a limit how much data they can use on the subscription they have for their mobile phone. It does it by having the ISP's provide Wikipedia for free.

There are other tools like Kiwix, it provides Wikipedia off line. Typically it is used in more formal settings like schools.

They are different tools they are both splendid and they both deserve praise and support.

Some say Wikipedia Zero is used largely by those who are already relatively wealthy and educated. The argument used is: "They use Wikipedia in English". This is a wonderful argument; it exposes the bias of Wikipedia for English and at most it is a perverse result.

Having people use Wikipedia in English does not mean that they do not benefit from Wikipedia Zero. If you want to know if Wikipedia Zero fulfils its goal of enabling people to read Wikipedia, check out the increased use of Wikipedia for those countries. The tool does what it is designed to do; enable access.

When the argument that more people should be able to access Wikipedia in their mother tongue, it means that the current bias for English is to be offset by investing in content in these other languages. <grin> These high horse arguments against Wikipedia Zero can be countered by un-orthodox approaches like paid editing and automated content </grin> Let the heated arguments commence !
Thanks,
      GerardM

#Kiwix - getting #Labs ready for the #Wikipedia big time

Offline #Wikipedia received a big boost. It is updating monthly its images for most of the #Wikimedia projects. Most but not all. Emmanuel was asked to write up about his challenges and I am happy to share this with his permission. Developments like this make both Labs and Kiwix even more strategic to out goals.
Thanks,
       GerardM

Following Yuvi's and Andrew's invitation, I write this email to explain what I want to do with Labs and share with you my first experiences. 

== Context == 

Most of the people still don't have a free and cheap broadband access to fully enjoy reading Wikimedia web sites. With Kiwix and openZIM, a WikimediaCH program, we have been working on solutions for almost ten years to bring Wikimedia content "offline".
We have built a multi-platform reader and have created ZIM, a file format to store web site snapshots. As a result, Kiwix is currently the most successful solution to access Wikipedia offline. 

== Problem == 

However, one of the weak point of the project is that we still don't achieve to generate often enough new fresh snapshots (ZIM files). Generating ZIM snapshots periodically (we want to provide a new fresh version each month) of +800 projects needs pretty much hardware resources.
This might look like a detail but it's not. The lack of up-to-date snapshots brakes many action within our movement to advert more broadly our offer. As a consequence, too few people are aware about it reported last Wikimedia readership update. An other side effect is that every few months, volunteer developers get the idea to build a new offline reader based on the XML dumps (the only up2date snapshots we provide for now), which is near to be a dead-end approach. 

== Goal == 

Our goal with Labs  is to have a sustainable and efficient solution to build, one time a month, new ZIM files for all our projects (for each project, one with thumbnails and one without). This is at the same time a requirement for and a part of a broader initiative which has for purpose to increase the awareness about our "offline offer". Other tasks are for example, storing all the ZIM files on Wikimedia servers (we currently only store part of them on download.wikimedia.org) and improve their accessibility by making them more visible (WPAR has for example customised their sidebar to provide a direct access 

== Needs == 

Building a ZIM file from a MediaWiki is done using a tool called mwoffliner which is a scraper based on both Parsoid & MediaWiki APIs. mwoffliner, after scraping and rewriting content, store them in a directory. At the end, the content is then self-sufficient (without online dependencies) and can be then packed in one step in a ZIM file (using a tool called zimwriterfs).
To run this software you better have:

• A little bit bandwidth

• Low network latency (lots of HTTP requests)

• Fast storage

• Pretty much storage (~100GB per million article)

• Many cores for compression (ZIM, ZIP and picture optimisation)

• Time (~400.000 articles can be dumped per day on a machine)

My guess is that we need a total of around a dozen of VMs and 1.5 TB of storage. 

== Current achievements == 

We have currently 3 x-large VMs in our "MWoffliner" project:
With them we are able to provide, one time a month, ZIM for all instances of Wikivoyage, Wikinews, Wikiquote, Wikiversity, Wikibooks, Wikispecies, Wikisource, Wiktionary and a few minors Wikipedias.
Here are a few feedbacks about our first months with Labs:

• Labs is a great tool, it's fully in the Wikimedia spirit and it works.

• Support on IRC is efficient and friendly

• We faced a little bit instability in December but instances seem to be stable now

• The Documentation on wikitech wiki seems to be pretty complete, but the overall presentation is to my opinion too chaotic and stepping-in is might be easier with a more user-friendly presentation.

• Mediawiki Sementic & OpenStackManager sync/cache/cookie problems are a little bit annoying

• Overall VM performance looks good although suffering from sporadic instabilities (bandwidth not available, all the processes stuck in "kernel time", slow storage).

In general, Labs does the job, we are satisfied and think this is an adapted solution to our project. 

== Next steps == 

We want to complete our effort and mirror the biggest Wikipedia projects. Unfortunately, we have reached the limits of a traditional usage of Labs. We need more quota and we need to experiment with the NFS storage because an x-large instance in not able to mirror more than 1.5 millions of articles at a time. How might that be made possible?

#Wikimedia & Project #Gutenberg - the sum of all knowledge

"To share in the sum of all knowledge" is the vision of the Wikimedia Foundation. The Swiss chapter does understand this really well. It has adopted Kiwix, an off line reader for content that is published in the ZIM format.

Project Gutenberg is a well established organisation dedicated to the digitisation of books. Its catalogue of 50.000 public domain books is now available to everybody, everywhere and offline as well.

Thanks to a hackathon, all books are now available in the ZIM format, you can search in all the books at the same time. The best news is that not only has this work been done for a first time, it is build in such a way that it can be easily repeated.

Future deployments may include all the books of Wikisource, books from other sources and even copyrighted works as well. The point of Kiwix is that it is an enabler, it allows for the dissemination of knowledge and to achieve THAT is what our aim is.

Congratulations to the Swiss Wikimedia chapter for providing the sustained support of this valuable project.
Thanks,
       GerardM

#Kiwix - the interview

#Kiwix is the tool that allows you to read the content of a Wiki offline. It has been developed with Wikipedia in mind but is equally usable for Wikisource or Wikibooks. I am really happy to have interviewed Emmanuel who knows all the ins and outs of this wonderful piece of software.
Thanks,
     GerardM


What is Kiwix and what is it used for
Kiwix is a software that wants to enable people to read Web contents without internet connection. It's a reader which works with ZIM files containing all the content. It's used to access Wikipedia offline, by reading pre-packaged Wikipedia ZIM files. It's mainly used by people who want to have an encyclopedia, but are too poor to have access to the internet. It's also used,for example, by travelling people (plane, ship, train), prisoners and students at school. 

Can you tell us something about its popularity
We have users all over the world and the audience is increasing quickly: we have had 25.000 downloads of Kiwix in May. 

In how many languages is Kiwix supported 
Thank to the Translatewiki Web site andits community, Kiwix is localised in more than 80 languages. We also provide content (ZIM files), mainly Wikipedia, in around 25 languages. But we want to do more: thanks to a grant of WMCH soon we will offer ZIM files of Wikipedia in all languages. 

How do you support languages written in scripts like Malayalam, or Tibetan
Contents are Web contents and Kiwix itself is a sort of browser getting the Web pages from the ZIM file instead of the Web. So, we do not have special handling in Kiwix itself to render the contents. Everything should be well organised in the ZIM file, for example by using Web fonts. But, from the Kiwix fulltext search engine point of view, this is challenging. Natural languages have a lot of particularities. Kiwix uses the Xapian search engine and tries to integrate CLucene. We do our best with them to offer the smoothest user experience possible. 

Do you provide fonts with Kiwix for the languages that use these scripts
The Wikipedia ZIM files we are preparing still do not provide the Web fonts. Already for a few months, the integration of Web fonts has been a part of the Wikimedia projects, so we have to fix that ASAP, this is not a big challenge.

For some languages like Chinese and Serbian, we show the content in two scripts ... Can Kiwix do this as well ?
Kiwix does not provide any transliteration tool for now, but all the technology is already there in the soft. We use a powerful unicode library called ICU (http://site.icu-project.org/) which can do that. We
want to use it to allow users to do custom transliterations. C++ developer wanted there!

Kiwix uses the OpenZIM format ... can you tell us more about this format
The format is called ZIM. There is a volunteer driven project called openZIM  created a few years ago to specify the format and develop a standard library. The ZIM format allows to put millions of contents together, to compress a part of them, and add Metadata. In the end, you get only one file, which is, at the same time, extremely compressed and allows a constant and quick random access.

Nowadays, many publications are in the EPUB standard ... can Kiwix handle this as well
Kiwix is not able to deal with EPUB, but in the future it will. We think EPUB & ZIM format are complementary and we want Kiwix being able to perfectly deal with EPUB. Our plan is to integrate "Monocle" to do that. Also there, developers are wanted.

How do people find content available for Kiwix
Kiwix has its own content manager, so you can download content from Kiwix itself. But you may also download the ZIM file from the Kiwix Web site (http://www.kiwix.org) or using the Mediawiki Collection extension.

In the future, we want to have a platform (something like Itunes) to offer  really easy to find and download contents (both ZIM and EPUB files). We have started a project in that perspective. We need your support! 

What is your biggest challenge at this moment in time with Kiwix
Building Kiwix-mobile for Android. We will have a first release in autumn. But we have many other projects running at the same time and others for which we still need volunteers.

Thank you
Emmanuel

#wmdevdays - finished a book a source, then what?

Both #Wikibooks and #Wikisource do a terrible job promoting their finished product. The Wikibookers and Wikisourcerers move on to the next book or source. A new source may get a moment of glory as a featured text or as a new text. A new book may be featured.

The problem is not so much with the Wikibooks and Wikisource projects, it is with what these projects actually are. They provide a workflow for the transcription, the proofreading and the final touches for the creation of digitised books and sources. As long as these are works in progress, they are not the finished product the general public is looking for.

The finished product of both projects is beautiful, lovable and deserves attention. The finished products deserve the attention of the public, they should be provided it in formats like EPUB and ZIM and it should be really easy for people to find and use them.

It is important to bring the best of what Open Content has to bring in sources and books to the public. Finding a big public will motivate many of our volunteers and it may bring us more volunteers. As we publish more titles, our projects gain relevance. As the final product becomes a more polished product people will love what we do. We already do.
Thanks,
     GerardM

Preferential treatment

The latest #translatewiki.net localisation rally had a twist. To qualify, the localisation for the Kiwix and the Wikipedia Mobile applications had to be completed for the language before MediaWiki messages were accepted.

Our thinking was: the most used messages are typically already complete and at this time growth of our content is very much taking place off-line or on a mobile phone.

For mobile phone use we have statistics that show a growth of 144% compared with last year December. Our support for mobile phones and the support for different writing systems on mobile phones is improving. As more people buy increasingly sophisticated mobiles, this growth may even increase.

During the localisation rally 30 additional languages were completed for Kiwix bringing the total to 65. For Wikipedia Mobile we now support 68 languages completely. We hope that our December 2011 rally sets the stage for a wonderful 2012 with even more people sharing in the sum of all knowledge.
Thanks,
     GerardM

#Localisation rally for #MediaWiki and #Kiwix

The #translatewiki.net localisation rally is finished. The results particularly for Kiwix are great. Thanks to the addition of 2,672 messages Kiwix is now fully localised in 53 languages. This makes it much easier for people to use our content off-line.

When you read the announcement of the results, it is particularly smaller languages that have benefited with the exception of traditional Chinese. When you look at the statistics, there are more languages that could have benefited from the rally and/or from more localisations.

This time the benefit to translators is EUR 37.50. The benefit to the users of the 20 languages however is where this rally makes a real difference.
Thanks,
       GerardM

#Localisation rally for #MediaWiki and #Kiwix

The Dutch #Wikimedia chapter sponsors #translatewiki.net regularly to run a translation rally. Such a rally serves multiple purposes;

• to bring the necessity of localisation to the front
• to support the localisation of MediaWiki and Kiwix
• to make the MediaWiki projects more usable

Internationalisation and localisation together make it possible for software to be used by people who do not understand the default language used for the User Interface of software. When these two factors are an integral part of software development like they are for MediaWiki, it becomes possible to support Wikipedia in over 270 languages.

The translatewiki rallies give an additional incentive to our localisers to do a little bit more. Many languages, including big ones like Spanish or Hindi can do with more effort. Both languages are used as "fall back" languages when the localisation of a language is not adequate but they need to be complete to do their job.

MediaWiki does gain relevant functionality over time; off line support with Kiwix and mobile telephone support now integral to MediaWiki are recent examples. This support is not restricted to Wikipedia, it is available to Wikimedia and non-Wikimedia projects alike.

You can join in the fun of the rally, you can make the user interface in your language more usable. The info can be found here but all your contributions are welcome for any and all the projects we support at translatewiki.
Thanks,
        GerardM

There are few messages to localise for #Kiwix

Kiwix can be localised at #translatewiki.net. and it started with 14 languages already localised. Two days later, 18 languages are completely localised and for 6 languages the localisation is underway.

With only 79 messages to localise, there is not much effort needed to localise Kiwix. Given that it is the client that does support the openZIM format, it helps support the Wikimedia Foundation in its effort to have a credible off-line strategy.

Check out the status, and help us to create the best off-line experience for your language.
Thanks,
       GerardM

#Kiwix is preparing for #translatewiki.net

At the Berlin hackathon,we talked with the Kiwix developer and he indicated that making Kiwix ready for localisation at translatewiki.net is one of his priorities.

As it one of those jobs that just does not go away, it is at some stage easier to give it a whirl. The good news is that a first iteration has found its way to translatewiki.

It is now a matter of making sure that new messages and localisations can find its way to and from Kiwix before the translatewiki community can work its magic.

More news when we have it :)
Thanks,
     GerardM

#Kiwix loves #WebFonts

When Kiwix targets a website for off line presentation, it takes all the attributes of that website in order to show it as is.

When a website does not use web fonts like Wikipedia at the moment does, the result of the exercise will also be determined by the quality of the support for a script on the local host. This is often patchy.

When Wikipedia installations use WebFonts, it will use the fonts selected as default for a language. Some people will prefer a different font for reasons best known to them.

The use of fonts supplied with Kiwix is being considered at the moment. When they do, they may choose to adopt all the fonts collected to support Wikipedia as they will all be freely licensed.
Thanks,
     GerardM

#Okawix makes it a two horse race

Going from an announcement to committed code makes Okawix the second off line program that supports openZIM. At this moment it is only the Linux client that has its initial code, but Okawix has the benefit of localisations in many languages.

When the software is stable for all the platforms, it will be interesting to see if the Wikimedia Foundation will acknowledge Okawix and give it the same status of that other open source product that supports openZIM, Kiwix.

In the end, the pudding is in the eating and when all the other things are equal, the difference will be in the quality and number of the localisations. Okawix has the advantage that is localised at translatewiki.net while Kiwix is localised using a text editor. Given that for many languages the localisation is not done from English and that there are no utilities that make the life of a localiser easy, Okawix has the best cards.

To be fair, we want a two horse race and handicaps are at the moment evenly divided. The race would be more interesting without handicaps; the Kiwix localisations deserve proper tooling and Okawix deserves support from the WMF.
Thanks,
      GerardM

Localisation for #Kiwix

Now that the #Wikimedia Foundation has decided to concentrate on openZIM for off line readers, Kiwix has gained a lot of prominence because it is still the only software that uses it.

Functionality to make a selection of articles for off line reading is being developed by PediaPress. This will be done by extending the Collections extension.

The good news is that work is now under way to internationalise Kiwix. This will make it possible for Kiwix to be localised at translatewiki.net.

As the presentation of data in the openZIM format is separate from the data, other off line readers like WikiReader or OkaWix have the option to adopt this format as well. When they do, they will profit from the work by the Wikimedia Foundation as well and when they do there will be a healthy competition for the hearts and minds of our off line readers.
Thanks,
       GerardM