Tuesday, July 31, 2012

Learning a new #script .. #Arabic

As I am learning a new language with a script that is new to me, I find the Internet not yet the resource it is in languages for the Latin script.

There are several obstacles; I have to configure my computing devices for the Arabic script and keyboard. I have to find the characters on my keyboard and only when I do will I get the facility that I need.

The one thing that would make my day is to have a typing tutor that is ready for an international public. This means that the instructions are available in multiple languages for the same input method for the same language. When these parts are separated from the code, there are three specific parts.
  • the user interface
  • the exercises
  • the input method or keyboard layout itself
This allows the same software to be used for multiple input methods and scripts / languages. The software can be localised at translatewiki.net and the typing tutor can be distributed with the input methods themselves. 
Thanks,
      GerardM

Wednesday, July 25, 2012

Multilingual cluefull bots


The #BBC writes that #Wikipedia without its bots is doomed. It is a good story, a happy story and well worth a read. Wikipedia as you know is not only the English language Wikipedia, there are over 280 Wikipedias in different languages. They are not all created equal; most of the smaller Wikipedias have less than 100.000 articles, a small community of editors and their own issues with people who think it funny to write penis wherever they can.

The BBC article writes about "Cluebot NG" and is said to reside on a computer somewhere. It would be a great project to move Cluebot NG to a Wikimedia server and make instances for all the other languages. First for the 40 something languages with more than 100.000 articles and when its lessons are learned there may be scope to expand.

There is a committee supervising the bot activities, it will be wonderful when it has the potential to expand its expertise as widely as possible.
Thanks,
       GerardM

Tuesday, July 24, 2012

Levels of competence for #Arabic in #Wikipedia III

Tools like #Google #Chrome use an engine to do the rendering for them. For Chrome it is Webkit. It is responsible to connect two items that are in separate HTML elements and make sure that the rules for the Arabic script are addressed.

What started as an observation in MediaWiki became a Chrome bug and is now a Webkit bug.

Webkit is used in multiple browsers.. This makes a fix relevant for Apple's Safari browser as well. The one question left is what will it take to bring a fix in production within those browsers in the near future.
Thanks,
       GerardM

Sunday, July 22, 2012

Levels of competence for #Arabic in #Wikipedia II

It does matter what browser you use. Google's Chrome browser is the reason why the Arabic Babel templates.

If you use Chrome to browse Arabic websites, you want to add your vote to fix this bug. You vote by marking the star that is at the bottom of the page.

I am really pleased how quickly this issue was analysed. It is now for Google to support its users who use languages in the Arabic script.
Thanks,
      GerardM

Levels of competence for #Arabic in #Wikipedia

As my competence in the Arabic language is growing, I added Arabic in my Babel information. To my amazement what I read is wrong. The "ba" is written on its own while it should be connected with the following "aliph". It is one of those peculiarities of the Arabic script that a character like the "ba" is expressed depending on its position in a word.

I had a look at translatewiki.net to understand why it is wrong; as you can see the "ba" or ب is 
connected to a wiki link and the word is not formatted after what is in the variable.

It is fun to learn other languages. What is really amazing is that I can find glaring issues like this one.
Thanks,
     GerardM

Thursday, July 19, 2012

European cranes want to be free

  

This wonderful video of European crane courtship is currently copyrighted and not freely licensed. The video comes with a button that allows you to embed it in a website and its article has a button where you can order the video.

The video is on the website of Stichting Natuurbeelden and they made a nice offer; 50 of their videos will become available under the CC-by-sa license. There are many great videos to choose from and for all kinds of reasons, the initial offering is to choose at most 50 of them. 

Organising this selection will not be easy and the relevance of some videos is very much in the context of nature in the Netherlands. European cranes are breeding again in the last few years. When the fifty videos are selected and when they are used in our projects, these videos will be seen quite often. At the moment when I write this, the number of views for this video is only 136.
Thanks,
     GerardM

Saturday, July 14, 2012

Can everybody read #Wikipedia?

A lot of effort goes into making "Wikipedia the encyclopaedia everybody can edit". The result is wonderful; there are Wikipedias in over 280 languages, a big effort is under way to make editing even easier and as so many people do edit, it became a rich almost authoritative resource. When Wikipedia goes off-line, students despair.

People do read Wikipedia, it is very popular but the notion that Wikipedia is hard to read is not really considered. Take for instance today's featured article.There are too many words on a line. For many people this makes it hard to read, some give up on an article or on Wikipedia.

Yes, you can change the way content is displayed on a computer screen. The problem is that people who have problems read websites in the default format.


The Wikimedia Foundation does have the expertise to consider these issues. The people who do have already too much on their plate to support the current software development. However, the proof of the pudding that is Wikipedia is in people READING its content and that makes this a key concern.
Thanks,
       GerardM

Tuesday, July 10, 2012

Something positive to say about #Apple

Apple is innovative, it routinely adds new functionality and quality to its products. Many people love Apple products and pay a healthy premium for the latest and greatest.

One of the recent innovations goes by the name of "retina display". It is a high resolution screen of such quality that the human eye does no longer see individual pixels. Innovations like the retina display add demand for high quality images and, it does stimulate the use of high quality images and the use of SVG or scalable vector graphics in WMF projects like Commons and Wikipedia.

The improved technology used in Apple hardware stimulates quality improvements at content providers. In this way Apple stimulates a healthy and innovating content ecosystem. As other hardware suppliers are continuously catching up with the leader of the pack, there is real value in buying Apple.

I did  it, nothing negative to read in this post about Apple.
Thanks,
        GerardM

#GLAM - About recognition

 Left Hand Bear, Oglala chief
This years #Commons picture of the year contest was different from last years. The many old images that were so lovingly restored and featured the Commons main page were not there any more.

A thread on the mailing list reminded me about all the hard work that gives images of the past a new lease of life. The image of Left Hand Bear, the Oglala chief is used a lot. As you can see below it is even used to make ties, mugs and buttons.

The image of Left Hand Bear has been lovingly restored by Adam Cuerden. The original of this image is at the Library of Congress and I owe a debt of gratitude to both Adam and the LoC.

Adam restored an image preserved at the Library of Congress. Knowing this, I am sure that this is indeed an image of Left Hand Bear. The image is obviously in the public domain and as such I am not required to acknowledge either the LoC or Adam. I may put the image on mugs, ties and buttons and sell them.

For both Commons and Wikipedia, acknowledging the LoC and Adam bring important benefits. Acknowledging the LoC provides provenance of the image, this is the equivalent of providing a source to a fact. Acknowledging Adam links the much improved image to the original. It recognises Adam for his work.

Acknowledging the LoC and Adam IS a best practice. It is a best practice promoted by organisations like Europeana. It is a best practice that is not a requirement, it is however something that we should aspire to.
Thanks,
      GerardM

Monday, July 02, 2012

#Kiwix - the interview

#Kiwix is the tool that allows you to read the content of a Wiki offline. It has been developed with Wikipedia in mind but is equally usable for Wikisource or Wikibooks. I am really happy to have interviewed Emmanuel who knows all the ins and outs of this wonderful piece of software.
Thanks,
     GerardM


What is Kiwix and what is it used for
Kiwix is a software that wants to enable people to read Web contents without internet connection. It's a reader which works with ZIM files containing all the content. It's used to access Wikipedia offline, by reading pre-packaged Wikipedia ZIM files. It's mainly used by people who want to have an encyclopedia, but are too poor to have access to the internet. It's also used,for example, by travelling people (plane, ship, train), prisoners and students at school. 

Can you tell us something about its popularity
We have users all over the world and the audience is increasing quickly: we have had 25.000 downloads of Kiwix in May. 

In how many languages is Kiwix supported 
Thank to the Translatewiki Web site andits community, Kiwix is localised in more than 80 languages. We also provide content (ZIM files), mainly Wikipedia, in around 25 languages. But we want to do more: thanks to a grant of WMCH soon we will offer ZIM files of Wikipedia in all languages. 

How do you support languages written in scripts like Malayalam, or Tibetan
Contents are Web contents and Kiwix itself is a sort of browser getting the Web pages from the ZIM file instead of the Web. So, we do not have special handling in Kiwix itself to render the contents. Everything should be well organised in the ZIM file, for example by using Web fonts. But, from the Kiwix fulltext search engine point of view, this is challenging. Natural languages have a lot of particularities. Kiwix uses the Xapian search engine and tries to integrate CLucene. We do our best with them to offer the smoothest user experience possible. 

Do you provide fonts with Kiwix for the languages that use these scripts
The Wikipedia ZIM files we are preparing still do not provide the Web fonts. Already for a few months, the integration of Web fonts has been a part of the Wikimedia projects, so we have to fix that ASAP, this is not a big challenge.

For some languages like Chinese and Serbian, we show the content in two scripts ... Can Kiwix do this as well ?
Kiwix does not provide any transliteration tool for now, but all the technology is already there in the soft. We use a powerful unicode library called ICU (http://site.icu-project.org/) which can do that. We
want to use it to allow users to do custom transliterations. C++ developer wanted there!

Kiwix uses the OpenZIM format ... can you tell us more about this format
The format is called ZIM. There is a volunteer driven project called openZIM  created a few years ago to specify the format and develop a standard library. The ZIM format allows to put millions of contents together, to compress a part of them, and add Metadata. In the end, you get only one file, which is, at the same time, extremely compressed and allows a constant and quick random access.

Nowadays, many publications are in the EPUB standard ... can Kiwix handle this as well
Kiwix is not able to deal with EPUB, but in the future it will. We think EPUB & ZIM format are complementary and we want Kiwix being able to perfectly deal with EPUB. Our plan is to integrate "Monocle" to do that. Also there, developers are wanted.

How do people find content available for Kiwix
Kiwix has its own content manager, so you can download content from Kiwix itself. But you may also download the ZIM file from the Kiwix Web site (http://www.kiwix.org) or using the Mediawiki Collection extension.


In the future, we want to have a platform (something like Itunes) to offer  really easy to find and download contents (both ZIM and EPUB files). We have started a project in that perspective. We need your support! 

What is your biggest challenge at this moment in time with Kiwix
Building Kiwix-mobile for Android. We will have a first release in autumn. But we have many other projects running at the same time and others for which we still need volunteers.

Thank you
Emmanuel

Saturday, June 30, 2012

#ImpactOCR - Digital publications and the national libraries

According to the #ISBN standard, the "format/means of delivery are irrelevant in deciding whether a product requires an ISBN". However, it is often assumed that a publication requiring an ISBN number is a commercial publication. In the USA and the UK for instance you have to buy your ISBN number or bar code while in Canada they are free because Canada stimulates Canadian culture.

When a standard is not universally applied, it loses application. When all publications are not registered a national library will have to maintain its own system when it is to collect a copy of all publications. As a result the ISBN is dysfunctional as a standard because it does not function as a standard.

When the Wikisourcerers finish the transliteration of a book, it deserves an ISBN number and, national libraries should be aware of these publications. This recognises and registered the work done in the Open Content world. When these books are registered, all the Open Content projects may know that they can concentrate on another book or source.

As the ISBN does not register all publications, it does not do what it is expected to do; function as a standard. 
Thanks,
       GerardM

Friday, June 29, 2012

Learning another language II

Learning to vocalise #Arabic can be done on the Internet. There are websites like Mount Hira where you find Arabic to the right, a transliteration to the left with an explanation in English underneath. What really helps is that you can listen to the recitation of a surah by line.

It is great but there is room for improvement. Improvement that can happen at Mount Hira but also at a Wikisource.
  • Show all the text as text and not as graphics
  • Allow for the Arabic text to be shown in fonts representing different writing styles
  • Allow for the explanations to be shown in a language that can be selected
Learning to vocalise what you read is one use case, learning Arabic is another and learning the Koran is a third. Wikisource has the potential to be a place for all three objectives.

There are always people who are interested in reading the source documents about a religion, any religion. The great thing of Wikisource is that you can include the wikilinks explaining the terms that are ambiguous or obscure.

Typically these source documents are readily available and out of copyright. Including them in Wikisource will gain it more public. Wikisource is an obvious place because it has a reputation to keep up; the reputation that original documents are original.
Thanks,
       GerardM

The Hebrew #Wikipedia is bigger than all of #Wikisource

When you compare the statistics for Wikipedia and Wikisource, the Wikipedia in Hebrew gets more eyeballs than all Wikisources combined. When you consider the potential public for both projects, the public for Wikisource is bigger by orders of magnitude.

On the main page of the English Wikisource it says: "We now have 783,122 texts in the English language library". It is less clear how many of these texts are in a final form; meaning ready for casual readers. The pages linked to the proof read tool indicate 691 completed pages and 3588 pages ready for proofreading. This does not imply that there are 691 text that are ready.

For a reader, the sources at Wikisource are works in progress. A text featured on the Wikisource main page, like the Celtic Fairy tales, is very much a Wiki page. The presentation of the text and illustrations is very much accidental. It is a bit sad is, but the pages where easier to read when they still needed to be proofread.


As you can see, the long lines below are not easy to read and the font used in the original book made the text more legible.


The Hebrew Wikipedia is easy to read. We can and should do better for Wikisource.
Thanks,
    GerardM

Thursday, June 28, 2012

#ImpactOCR - A #font using #unicode private use characters

When really old historic texts are digitised and OCR-ed, the images of letters found are mapped to the correct characters. Characters are defined in Unicode and when a character is NOT defined, it is possible to define them in the "private use" space.

As part of the Impact project, really old texts have been digitised, texts in many languages. At the recent presentation it was mentioned by two speakers that there were characters used in the Slovenian and Polish language that are not (yet) defined in Unicode. As part of their project, the missing characters were defined in the Unicode private use area and the scanning software was taught to use them.

With the research completed, with the need for all these characters and their shape defined, it will be great when these characters find their way in Unicode proper. When the code points for the missing characters are defined and agreed, the OCR software can learn to recognise the characters at the new code points, a conversion program can be written for the existing texts and it will be more inviting to include these characters in fonts.

Now that the project is at its end, it is the right moment to extend the Latin script in Unicode even further.
Thanks,
     GerardM

#ImpactOCR - Citing a #newspaper

#Wikipedia has this rule: "Citation needed". Much of the news is first published in newspapers. When a citation is needed about something that happened, it is in the newspaper where you will find it mentioned and there may be many chronological entries on the same subject describing how something evolves.

A lot of research and development has gone in the optical character reading of newspaper of the Impact project. As this project has ended and has evolved into a competence centre, its last conference was very much a presentation of what the project achieved.

From my perspective, it produced a lot of software much of it open sourced and all of it is implemented and embedded in the library, archive and research world. It is a world that finds its public for the work done in the Impact project very much in the research world. The general public can benefit as much, what has to be clear is how it could benefit.

Though Europeana newspapers some 10 million newspaper pages will be made accessible. These pages are scanned and to make them really useful they undergo optical character recognition. This is exactly where the Impact project has its impact; as the OCR technology improves, more words are correctly recognised and consequently more content of the newspapers can be discovered.

The results can be improved even further when the public helps train the OCR software recognise characters for specific documents. As citing sources for Wikipedia is an obvious use case for historic newspapers, there are many people who are willing to teach OCR engines to do a better job. For those articles that are found to be particularly useful, proofreading can improve the results even further.

With a public that is involved in improving the digitised and OCR-ed texts everybody will be a winner including the scientific research on these texts.
Thanks,
      GerardM

Monday, June 25, 2012

Learning another language

Learning to read #Arabic or more correctly Classical Arabic or "al fusha" is fun. I enjoy it for the challenge it provides, it helps because the many transliteration schemes suck and often confuse more than help.

When I started on this road to learn Classical Arabic, I was told that there is only one Arabic and I am also told that when you speak Classical Arabic in Arabic speaking countries, you are thought to be weird and often you find yourself not understood. I do not point out as often that the second assertion negates the first and consequently there are many different Arabic languages. I do not yet understand to what extend the Koranic Arabic language is different from what is labelled "standard" Arabic language. 

At this time I am learning to pronounce fully annotated texts. When you are used to Latin script, one of the lessons is that a space is not necessarily the dividing line between words. This is hard because it clashes with how you have learned to perceive a word at a glance. Al Arabiya or العربية shows two spaces but for me it is one word. To complicate it further, the vowels are missing and you need to know the Arabic grammar and vocabulary to know how to pronounce it with certainty.

Add to this the different ways Arabic is written and printed and you will appreciate learning the Arabic script for the intellectual challenges it presents.
Thanks,
     GerardM

Monday, June 18, 2012

#Batak - getting a #script ready for the #Internet

Several languages from Sumatra, #Indonesia were originally written in the Batak script. This script was encoded in Unicode and the waiting was for a freely licensed font. Thanks to a grant by the Wikimedia Foundation, a project is under way that will produce a font for Batak and will transcribe sources from Dutch museums like the Tropenmuseum.

For a script to be encoded into Unicode, a lot of research goes into describing the font and how it functions. For Batak you can find this documentation here.

When you read this, you find details like how the combination of a consonant and a vowel is expressed in the Batak script. When smart algorithms are used only the valid combinations will be expressed.

Given that multiple languages use the Batak script, the rules that are implemented in a font need to allow for every Batak language.

The first iterations of the Unicode font for Batak are being tested. When this is done, the font will be available and it will be possible to transcribe original Batak sources including a book on sorcery.
Thanks,
       GerardM



Saturday, June 16, 2012

Wikis waiting to be renamed

Many a #Wikipedia was created before  the language policy was created. One of the requirements of the language policy is that any request for a new Wikipedia is in a language that is recognised in the ISO-639-3 standard.

At that time several Wikipedias were created that were not recognised as a language. In the mean time several of these have been recognised as a language and as a consequence have their own code.

bat-smg -> sgs (wikipedia)
fiu-vro -> vro (wikipedia)
zh-classical -> lzh (wikipedia)
zh-min-nan -> nan (wikipedia, wiktionary, wikibooks, wikiquote, wikisource)
zh-yue -> yue (wikipedia)

With the deployment of  bug 34866, an important improvement has been realised; the content of the Wikis involved do now indicate correctly in what language they are written. This helps because there is now more content correctly available on the Internet.

It is a relevant step in the direction of giving many wikis the name we would like them to have.
Thanks,
     GerardM

Thursday, June 14, 2012

#Wikidata can get more facts out

The big discussions that are raging on mailing lists about the infoboxes that can be populated with data from Wikidata are amusing. Amusing because it has been made plain that this is not something that is planned for in the near future. Amusing because there is more to it.

Consider the following scenario; two Wikipedias do not agree on what source to use for a specific bit of information. The result is different data in the infobox. What is passed as a solution is to override things locally. Nice thought but it is the wrong approach; it is reasonable to assume that multiple Wikipedias opt for either option. Wikidata as a data repository is agnostic to such issues. It can happily store information from multiple sources and have multiple info boxes for the same category of subjects.

You may be able to override and maintain data on a local level. Doing so does not make it a best practice. Far from it, Wikipedia has this neutral point of view and Wikidata does change the rules. It has always been recognised that the NPOV is set by a language community and its prevailing wisdom is not necessarily neutral. The info boxes will not only bring consistency to the data, it will necessarily bring home the notion that facts have sources and many sources have a point of view.
Thanks,
      GerardM

Wikisourcery


Fairly often when you complain about issues Wikimedia, there is somebody who notices, who cares. This is the sourcery of the crowds, the magic of our communities. Compare the picture above with the picture below..


What a difference a day makes ..
Thanks,
        GerardM