#Kiwix - getting #Labs ready for the #Wikipedia big time

Offline #Wikipedia received a big boost. It is updating monthly its images for most of the #Wikimedia projects. Most but not all. Emmanuel was asked to write up about his challenges and I am happy to share this with his permission. Developments like this make both Labs and Kiwix even more strategic to out goals.
Thanks,
       GerardM

Following Yuvi's and Andrew's invitation, I write this email to explain what I want to do with Labs and share with you my first experiences. 

== Context == 

Most of the people still don't have a free and cheap broadband access to fully enjoy reading Wikimedia web sites. With Kiwix and openZIM, a WikimediaCH program, we have been working on solutions for almost ten years to bring Wikimedia content "offline".
We have built a multi-platform reader and have created ZIM, a file format to store web site snapshots. As a result, Kiwix is currently the most successful solution to access Wikipedia offline. 

== Problem == 

However, one of the weak point of the project is that we still don't achieve to generate often enough new fresh snapshots (ZIM files). Generating ZIM snapshots periodically (we want to provide a new fresh version each month) of +800 projects needs pretty much hardware resources.
This might look like a detail but it's not. The lack of up-to-date snapshots brakes many action within our movement to advert more broadly our offer. As a consequence, too few people are aware about it reported last Wikimedia readership update. An other side effect is that every few months, volunteer developers get the idea to build a new offline reader based on the XML dumps (the only up2date snapshots we provide for now), which is near to be a dead-end approach. 

== Goal == 

Our goal with Labs  is to have a sustainable and efficient solution to build, one time a month, new ZIM files for all our projects (for each project, one with thumbnails and one without). This is at the same time a requirement for and a part of a broader initiative which has for purpose to increase the awareness about our "offline offer". Other tasks are for example, storing all the ZIM files on Wikimedia servers (we currently only store part of them on download.wikimedia.org) and improve their accessibility by making them more visible (WPAR has for example customised their sidebar to provide a direct access 

== Needs == 

Building a ZIM file from a MediaWiki is done using a tool called mwoffliner which is a scraper based on both Parsoid & MediaWiki APIs. mwoffliner, after scraping and rewriting content, store them in a directory. At the end, the content is then self-sufficient (without online dependencies) and can be then packed in one step in a ZIM file (using a tool called zimwriterfs).
To run this software you better have:

• A little bit bandwidth

• Low network latency (lots of HTTP requests)

• Fast storage

• Pretty much storage (~100GB per million article)

• Many cores for compression (ZIM, ZIP and picture optimisation)

• Time (~400.000 articles can be dumped per day on a machine)

My guess is that we need a total of around a dozen of VMs and 1.5 TB of storage. 

== Current achievements == 

We have currently 3 x-large VMs in our "MWoffliner" project:
With them we are able to provide, one time a month, ZIM for all instances of Wikivoyage, Wikinews, Wikiquote, Wikiversity, Wikibooks, Wikispecies, Wikisource, Wiktionary and a few minors Wikipedias.
Here are a few feedbacks about our first months with Labs:

• Labs is a great tool, it's fully in the Wikimedia spirit and it works.

• Support on IRC is efficient and friendly

• We faced a little bit instability in December but instances seem to be stable now

• The Documentation on wikitech wiki seems to be pretty complete, but the overall presentation is to my opinion too chaotic and stepping-in is might be easier with a more user-friendly presentation.

• Mediawiki Sementic & OpenStackManager sync/cache/cookie problems are a little bit annoying

• Overall VM performance looks good although suffering from sporadic instabilities (bandwidth not available, all the processes stuck in "kernel time", slow storage).

In general, Labs does the job, we are satisfied and think this is an adapted solution to our project. 

== Next steps == 

We want to complete our effort and mirror the biggest Wikipedia projects. Unfortunately, we have reached the limits of a traditional usage of Labs. We need more quota and we need to experiment with the NFS storage because an x-large instance in not able to mirror more than 1.5 millions of articles at a time. How might that be made possible?

#Wikimedia #Labs - Risc analysis

Labs is a wonderful and successful project; more virtual machines are added all the time. More data is produced all the time and more people rely on it all the time.

Sounds good? It is!

From a management point of view it becomes increasingly problematic because for many of the most valuable Wikimedians it became a production resource and, as Labs is growing really quickly, it easily escapes the boundaries set earlier. Staffing, hardware it could all be better and it should all be better.

Having the best possible Labs will grow Labs even more. The best will outwit and outperform expectations. Classical budget think is a disservice to what we may achieve: share more information as widely as possible. One approach is to maintain a risc analysis of the services provided by Labs. It will help management to manage, to think and to use funds when the need and the justification is bigger than the budget

Today new virtual machines have been started that are starting to produce ZIM files based on the latest dumps. This will improve off-line reading of our projects a lot. The ZIM files will in future be and remain fresh..

This is just one day in the life of Labs...
Thanks,
      GerardM

Test #offline #Wikipedia on your #Android or #iPhone

#Okawix is completing its functionality for the best selling smart phones. Now they need people to test it for them so that they can pick off all the bugs.

Testing and localising the Okawix software are the perfect preparation for the next phase of its development, the support for openZIM. The realisation of this next phase will surely make the Wikimedia Foundation really happy because it will finally provide them with an application that supports Wikipedia content in the openZIM format with a user interface in the same language.
Thanks,
      GerardM

#Okawix will support #openZIM

Okawix will support openZIM. #Wikipedia off line reading is done best when a localised interface is important to you. It is the only application that can be localised that will support openZIM.

With the support of openZIM, Okawix supports two formats for texts; their original one  Zeno and openZIM which is more efficient in compressing text. The installed base of Okawix includes smart phones like Android and iPhone. As Android does not yet support lzma2 compression, Okawix will move to the openZIM format when this support is available.

Okawix can be localised at translatewiki.net and with zeno it is a tool that not only displays the content of your wiki but also can also have a user interface in your language.
Thanks,
       GerardM

Localisation for #Kiwix

Now that the #Wikimedia Foundation has decided to concentrate on openZIM for off line readers, Kiwix has gained a lot of prominence because it is still the only software that uses it.

Functionality to make a selection of articles for off line reading is being developed by PediaPress. This will be done by extending the Collections extension.

The good news is that work is now under way to internationalise Kiwix. This will make it possible for Kiwix to be localised at translatewiki.net.

As the presentation of data in the openZIM format is separate from the data, other off line readers like WikiReader or OkaWix have the option to adopt this format as well. When they do, they will profit from the work by the Wikimedia Foundation as well and when they do there will be a healthy competition for the hearts and minds of our off line readers.
Thanks,
       GerardM

#openZIM is not supported here

The off line reader for #Wikipedia will be based on openZIM. Now openZIM is one of several formats that are actively maintained for use in off line readers. One of them has been specifically developed for use with Indic languages and others like Okawix and WikiReader are internationalised and are localised in many languages.

A choice has been made for a tool that has no obvious qualifications to support the over 270 languages supported by Wikipedia.

The good news is that at translatewiki.net we are quite happy to support competing applications. As we reuse existing localisations for new applications, openZIM may even benefit from the work that has been done on off line readers.

So Wikimedia Foundation, it is not obvious why you support openZIM. OpenZIM is not supported here but that can be remedied when you care for supporting all the languages of Wikipedia.
Thanks,
     GerardM