Indian #politicians from #Kerala

The information on politicians from #India on the Malayalam Wikipedia is more extended for the members of the Kerala legislative assembly than on the English.. That is entirely reasonable and good.

It shows that English does not include the sum of all information that is available to us. It also shows that we need someone to help us translate the names of these fine people into English to make this available to you and me.
Thanks,
      GerardM

#Kiwix - getting #Labs ready for the #Wikipedia big time

Offline #Wikipedia received a big boost. It is updating monthly its images for most of the #Wikimedia projects. Most but not all. Emmanuel was asked to write up about his challenges and I am happy to share this with his permission. Developments like this make both Labs and Kiwix even more strategic to out goals.
Thanks,
       GerardM

Following Yuvi's and Andrew's invitation, I write this email to explain what I want to do with Labs and share with you my first experiences. 

== Context == 

Most of the people still don't have a free and cheap broadband access to fully enjoy reading Wikimedia web sites. With Kiwix and openZIM, a WikimediaCH program, we have been working on solutions for almost ten years to bring Wikimedia content "offline".
We have built a multi-platform reader and have created ZIM, a file format to store web site snapshots. As a result, Kiwix is currently the most successful solution to access Wikipedia offline. 

== Problem == 

However, one of the weak point of the project is that we still don't achieve to generate often enough new fresh snapshots (ZIM files). Generating ZIM snapshots periodically (we want to provide a new fresh version each month) of +800 projects needs pretty much hardware resources.
This might look like a detail but it's not. The lack of up-to-date snapshots brakes many action within our movement to advert more broadly our offer. As a consequence, too few people are aware about it reported last Wikimedia readership update. An other side effect is that every few months, volunteer developers get the idea to build a new offline reader based on the XML dumps (the only up2date snapshots we provide for now), which is near to be a dead-end approach. 

== Goal == 

Our goal with Labs  is to have a sustainable and efficient solution to build, one time a month, new ZIM files for all our projects (for each project, one with thumbnails and one without). This is at the same time a requirement for and a part of a broader initiative which has for purpose to increase the awareness about our "offline offer". Other tasks are for example, storing all the ZIM files on Wikimedia servers (we currently only store part of them on download.wikimedia.org) and improve their accessibility by making them more visible (WPAR has for example customised their sidebar to provide a direct access 

== Needs == 

Building a ZIM file from a MediaWiki is done using a tool called mwoffliner which is a scraper based on both Parsoid & MediaWiki APIs. mwoffliner, after scraping and rewriting content, store them in a directory. At the end, the content is then self-sufficient (without online dependencies) and can be then packed in one step in a ZIM file (using a tool called zimwriterfs).
To run this software you better have:

• A little bit bandwidth

• Low network latency (lots of HTTP requests)

• Fast storage

• Pretty much storage (~100GB per million article)

• Many cores for compression (ZIM, ZIP and picture optimisation)

• Time (~400.000 articles can be dumped per day on a machine)

My guess is that we need a total of around a dozen of VMs and 1.5 TB of storage. 

== Current achievements == 

We have currently 3 x-large VMs in our "MWoffliner" project:
With them we are able to provide, one time a month, ZIM for all instances of Wikivoyage, Wikinews, Wikiquote, Wikiversity, Wikibooks, Wikispecies, Wikisource, Wiktionary and a few minors Wikipedias.
Here are a few feedbacks about our first months with Labs:

• Labs is a great tool, it's fully in the Wikimedia spirit and it works.

• Support on IRC is efficient and friendly

• We faced a little bit instability in December but instances seem to be stable now

• The Documentation on wikitech wiki seems to be pretty complete, but the overall presentation is to my opinion too chaotic and stepping-in is might be easier with a more user-friendly presentation.

• Mediawiki Sementic & OpenStackManager sync/cache/cookie problems are a little bit annoying

• Overall VM performance looks good although suffering from sporadic instabilities (bandwidth not available, all the processes stuck in "kernel time", slow storage).

In general, Labs does the job, we are satisfied and think this is an adapted solution to our project. 

== Next steps == 

We want to complete our effort and mirror the biggest Wikipedia projects. Unfortunately, we have reached the limits of a traditional usage of Labs. We need more quota and we need to experiment with the NFS storage because an x-large instance in not able to mirror more than 1.5 millions of articles at a time. How might that be made possible?

The #Italian job; a perspective on #bias

The Italian #Wikipedia is really into registering death and place of death. This is why an initial bot-run by Amir resulted in more deaths in Italy then elsewhere. This bias was visualised by Vizidata, I blogged about it in the past.

Today Amir finished another run of his bot on the Italian Wikipedia and he increased the bias with over 100,000 edits.

The point is very much that Wikidata has a bias and it is based on the quality of the Wikipedia data depended on. Running a bot repeatedly will increase quality and by inference bias. People die in Italy, apparently not so much in the Netherlands.

Now who says this Italian bias is bad?
Thanks,
      GerardM

#Wikimedia Nederland - #Sebastiaan is leaving

Sebastiaan was project leader at the Dutch Wikimedia chapter. He did tons of great work, and made a point of registering activities with his camera. Of particular mention is his huge involvement in the GLAM area.

Sebastiaan will pursue his career elsewhere. His friendly cooperative attitude will be sorely missed.
Thank you,
       Gerard

#Diversity - the Lillian Smith Book #Award II

There are two ways of improving the content of Wikidata. It can be by adding large amounts of statements or by adding more details to existing data. As I was adding the details, I found that several award winners do not have an article. Adding them in Wikidata is easy and obvious.

Mr A.G. Mojtabai for instance received the award in 1986. Adding a red link in Wikipedia is not hard either. Thanks to the Redwd template, I linked him to both Wikidata and to Reasonator. One issue is that all these authors and the award are primarily known on the English Wikipedia. Consequently their work and relevance has at this time a limited public.

It would be nice when the presence of great information at Wikidata will lead to articles in other languages. The question is very much if it does.
Thanks,
     GerardM

#Hackathon - the #genealogy of Catharina-Amalia, Princess of Orange

Visualising the genealogy of person can be extremely interesting. The tool that was the best we had exists in Reasonator and for the Dutch Royal family it did not really work. Too many people are involved.

The approach of a new tool developed at the hackathon in Bern is really interesting. It limits itself to five generations and it shows pictures of the people. It is nice to see princess Catharina-Amalia.

It is also great to notice that you can have the same information in for instance Chinese or Russian. When you click on one of the persons in the genealogy, it will produce the genealogy for that person..

At this time the new tool is very much in development. It is great to show why hackathons are so relevant.
Thanks,
     GerardM

#Diversity - the Lillian Smith Book #Award

Lillian Smith who is obviously white, openly embraced controversial positions on matters of race and gender equality, she was a southern liberal unafraid to criticize segregation and work toward the dismantling of Jim Crow laws, at a time when such actions almost guaranteed social ostracism.

The Lillian Smith Book award honours those authors who, through their outstanding writing about the American South, carry on Smith's legacy of elucidating the condition of racial and social inequity and proposing a vision of justice and human understanding.

It is obvious that these writers are important as sources for the subject and consequently, registering them as award winners is important. This was done by harvesting the information from the article using Magnus's LinkedItems tool,

Obviously more can be done.

• including all her work in Wikidata; it does not need a Wikipedia article
• including all the works of the prize winning authors
• adding dates as a qualifier for the award winners
• complete the list of award winners
• work on similar awards

There is always more that can be done on a subject as relevant as this.

Thanks,

      GerardM

Harvest #Wikipedia categories

For many categories it is obvious what they should include. "Indiana State Senators" for instance will all be human. So when we know that an article is a human, we can safely deduce that they indeed hold or held a position as "member of the Indiana State Senate".  We can do similar things for alumni or members of a sports team.

When we harvest such categories regularly, Wikidata will become more inclusive than any Wikipedia. This is because we can harvest from similar categories from any Wikipedia.

We can, we should. harvest Wikipedia categories regularly. It will enrich Wikidata and we will become more aware of the full scope of the information held in all the Wikimedia Wikis.
Thanks,
      GerardM

#Wikimedia #Labs - Risc analysis

Labs is a wonderful and successful project; more virtual machines are added all the time. More data is produced all the time and more people rely on it all the time.

Sounds good? It is!

From a management point of view it becomes increasingly problematic because for many of the most valuable Wikimedians it became a production resource and, as Labs is growing really quickly, it easily escapes the boundaries set earlier. Staffing, hardware it could all be better and it should all be better.

Having the best possible Labs will grow Labs even more. The best will outwit and outperform expectations. Classical budget think is a disservice to what we may achieve: share more information as widely as possible. One approach is to maintain a risc analysis of the services provided by Labs. It will help management to manage, to think and to use funds when the need and the justification is bigger than the budget

Today new virtual machines have been started that are starting to produce ZIM files based on the latest dumps. This will improve off-line reading of our projects a lot. The ZIM files will in future be and remain fresh..

This is just one day in the life of Labs...
Thanks,
      GerardM

#Wikidata - #Alumni by university or college in #India

As one of the most populous countries in the world, it is no surprise that India has many universities. The alumni of Indian universities or colleges can be found through a Wikipedia category.

Whenever tools are down I have been adding these alumni to Wikidata. It seems obvious that not all universities and colleges are represented. It is certain that many alumni cannot be found in these categories. This is because there may be no article about them or they have not been included in the category.

It is relatively easy to do this for India given that English is the main language for subjects about India. For China, Russia and Japan it is not so easy. Someone else has to get involved as I do not know the languages.

All of Labs is down again. So this time my customary hyperlinks are sadly absent..
Thanks,
      GerardM

#Wikidata - the Stern–Gerlach Medal

The Stern–Gerlach Medal is one of many awards Wikidata knows about. Information is often available in a list within the articles. In some languages there are links to all those who received the award.

Having all the awards and all the people who received them in Wikidata is a massive undertaking. It can be argued that everyone who received an award has some notability..

Some people think that awards are not that important to categorise. Their way of thinking means that awards specifically relevant within a culture, a language become underrepresented. This is however an effect that diminishes in time.

It would be good when the lists were available to Wikipedias to use. When such lists become a service from Wikidata, it is easy to provide minimal information for the people that do not have an article yet. For best results it helps when all the associated labels are available.
Thanks,
      GerardM

Where people died; a perspective on diversity

A wonderful new view is available thanks to Vizidata. It shows where people were born and were people died. The data is from a Wikidata dump so it is sadly static. Given that it is from Wikidata, you can safely assume that the data also exists in a Wikipedia ...

Italy is well pronounced in this view. It is because a lot of effort went into extracting data from the Italian Wikipedia. It follows that all the people the Italians care for are included as well. The fun thing of a view like this is that it is a historic view of what Wikidata covers and does not cover..

Apparently hardly anyone died in Africa in all the centuries.
Thanks,
      GerardM

25.000 books, old books

When 25.000 books, books from the early days, English texts from 1473-1700 become available it is quite something. Many of these text are the earliest sources on many subjects in English.

All of them deserve to be registered in Wikidata, The most relevant question would be: how do we serve our public best. Yes, it starts with indicating that these books exist but it is easy enough to point people in the right direction. The direction where these books can be found to be read.

It seems obvious. When books are (finally) available under a free license, it is important for people to find them.
Thanks,
      GerardM

Rafik Tlili, member of the Constituent Assembly of #Tunisia

Mr Tlili is a Tunisian politician who died. What is refreshing is that there is at least one decent list of members of the current parliament and, as is fitting, it is in French. Without assistance of Google translate the articles are too difficult for me.

There is also a category; and it has a problem. It links the current members in French to every member of the Tunisian parliament. from a Wikidata point of view that is fatally flawed. It is however part and parcel of a category of subjects that is underdeveloped. Our Wikiverse does not really care about Farfarawayistan. Its problems is seen as the diversity that is in genders and while important, it easily ignores what is far far away. As you can see in the picture, there are a fair bunch of women in the Tunisian parliament.

Even people who research are interested in diversity. They want to know how diversity differs in different languages. Those different languages mean different cultures, Cultures that by and large are not really well known in our Wikipedias as they are far, far away. Consequently Wikidata does not serve them the data they need.

I am happy with the Tunisian list. It means that Tunisia is not longer as far far away.
Thanks,
       GerardM

CC-BY-SA; Creative Commons needs our support

The CC-BY-SA licenses are crucial to the Wikimedia movement. All but one of the Wikimedia Foundation projects are licensed with CC-BY-SA; Wikidata being the exception.

I find it astounding to learn that Creative Commons is in financial dire straits. As Wikimedians we are part of a world that is shaped by copyright law and the fight for free and fair license. When a crucial player like Creative Commons cannot take its role, it shows our weakness, It indicates that we are fighting a losing battle because our priorities are wrong.

Creative Commons deserves our support. We rely on Creative Commons.

It is one of those organisations that the WMF could do something special for. For instance a fund raiser on their behalf. <grin> WMF is good at that </grin> and in this way commit ourself more to free and fair licenses.
Thanks,
        GerardM

#Wikidata - Tokyo University of the Arts alumni

Mr Kenji Ekuan died. He was a Japanese designer who studied at the Tokyo University of the Arts. As can be expected, he was not the only one who studied there. There are categories in several Wikipedias informing us about them. The category on the Japanese Wikipedia has includes 642 alumni and for many of them there is no label in other languages.

It is no surprise either that these people refer to many items that have a label in Japanese and not in the languages people are familiar with. The automated description for Mr Munemoto Yanagi in Dutch for instance is "kunsthistoricus (*1917); Mainichi-Kulturpreis; kind van 柳宗悦 en 柳兼子 ♂". As more labels become available in Dutch, this information becomes easier to comprehend.

With every label that is added, all the associated descriptions are improved. Every item will be easier understood in Reasonator as well. Adding labels in Reasonator will provide you with instant gratification. Every statement of that item will show the new label.
Thanks,
     GerardM

#Wikidata - automated #descriptions are GIGO

The great thing about automated descriptions of Wikidata items is that they reflect what is there. The consequence is that when it is not good, it will show.

I discussed automated descriptions with my friend Amir and he pointed me to that "East Jerusalem" is not a "world heritage site". He is absolutely right; world heritage site is a qualification of whatever it is that has been recognised in this way.

Amir also pointed out that whenever labels are missing, automated description will give you something in another language. This is not a bug, it is a feature. All it takes is for someone to add the missing label and wherever it is used, the label will show in the "current" language in future. One alternative is NOT showing statements when they have no labels. Another alternative is to make it configurable if you want as much as possible or information only in the current language.

The point about automated descriptions stands; adding one statement may impact descriptions in all languages. Adding one label will impact all items that include the associated statement for that language.

Obviously an automated description is superior to no description. Arguably automated descriptions are superior to manual description because they reflect the item in any and all language.
Thanks,
       GerardM

#Wikidata - George Muchai member of the Kenyan parliament

Mr Muchai was assassinated. For parliamentarians to be killed in this way is shocking. It is not important if you agree with a politician, a journalist, anyone. You do not kill them; you argue their points, maybe.

The death of Mr Muchai was reported in the international press. Wikipedia does not have an article. When someone finds it relevant enough they may write an article about him. They may call it "the assassination of George Muchai" At least there is already a Wikidata item for him.
Thanks,
     GerardM

#Wikidata descriptions are not best of breed

In a Wikimedia mobile application they are going to use Wikidata descriptions. I really wonder why they did this because it is easy to recognise why it is substandard.

Take Mr Robert E. Hanson for instance. According to the description he is an "American politician", the description exists only in English and it does not recognise that he died.

Take Mrs Татьяна Михайловна Лебедева for instance; she died as well and there is not even a description.

At the same time there are descriptions available for both of them in any language based on the availability of statements and labels. For Mr Hansen it is "Amerikaans politicus (1947–2015) ♂" in Dutch and for Mrs Лебедева it is "Mens (1944–2015) ♀". These descriptions could easily have been in Arabic, Chinese, Zulu, Javanese or any of the other languages the WMF supports.

It has been obvious for a long time that descriptions are flawed and will not be useful in most languages supported by the WMF. It has been as well known that automated descriptions are more useful in more languages. The developers may have created a best of breed solution but it is still very much a dog.
Thanks,
      GerardM

#Wikidata - when qualifications fail

Mr Diego Betancur Álvarez is the son of a former Colombian president. He is the ambassador for Colombia to Australia and New Zealand. Being ambassador is a position that a diplomat may hold. It is similar to president to the United States which is a position for a politician.

It is quite normal that people do not understand such qualifications. It is however the essence of what Wikidata is about.

It is easy enough to make any and all humans who are known as ambassador a professional diplomat. It is easy enough to give them the position of ambassador as well. It just needs to be done.

Another approach would be to have one "ambassador to Australia" or any other country. The alternative is to rely on qualifiers.
Thanks,
     GerardM

PS Mr Betancur is very much alive.