Over the weekend I started work on a new project. As I've mentioned before, I want to make an app that will assist others in reading the encyclopedia. Of course, that means wikipedia rather than Britannica.
The first step involved processing the vast amount of data in wikipedia. I downloaded the bzip2 compressed 15 gb file containing all the textual data in wikipedia. I spent most of Saturday night and Sunday afternoon working on it, but I now have a program that processes that file (without needing to decompress it first) and outputs an xml file containing a stripped down version that contains only the introduction of each article, the id of the article, and an image from the article if possible. By my calculations, it should be able to process the whole file in about 30 hours.
There were plenty of obstacles. I needed to find a Java library that handles streaming a bzip2 file. I found one in the ant project. I needed to parse the xml in a streamed manor (no way that I could load a 100gb decompressed xml file into memory). That was easy enough. I've used Java StAX before.
Finally I needed to parse the mediawiki format, strip out all the data I wanted to ignore, convert it to a format that I could use in an android app, then find a useful image link from the article. This was tricky, but I found a library called Sweble that parses mediawiki format. It's poorly documented, but once I figured it out, it worked beautifully.
Mediawiki articles sometimes have image links in the description, sometimes that have them in a sidebar, sometimes they have it only in the main article, and sometimes they have no image at all. I can now handle most of the easy cases, but to save time I currently omit the body of the article from parsing. That means I have no image link for any article that has image links exclusively in the body of the article. I'll have to work on how to handle that without adding significant time to the parsing.
The next step is to get genre information and generate a pagerank so that I can determine what articles to prioritize. When someone is reading the encyclopedia, it'll be terribly boring if I only show them the most popular articles. They would include almost nothing but celebrities and current events. Some might be interested in that, but I want to split things into genres and then include only the most popular and/or important articles in each genre.
Importance is hard to judge though. Popularity is one way (judged by page hits). Another way is page rank (a wikipedia article linked to by lots of other wikipedia articles has a higher page rank than one that's never linked to). Another way is centrality. If you plot a graph of how everything is linked, some articles will show up more centrally whether or not they are linked directly to more articles.
I wasn't able to find a freely available source of page rank data for wikipedia, so I'll have to calculate it myself. That'll be my project next. I'll parse the wikipedia data, output a list of pairs indicating which articles link to which other articles. Then I'll generate page rank data from that.
I'm hoping to find genre information, but I may need to derive that from information in the article itself like the info boxes. Then, for articles without info boxes, I could probably determine genre by finding how strongly they are linked to articles of known genre. That'll probably be a big project. I'm still hoping I can find sources of this data so that I can make use of it, but if it's not available, it should be fun to figure it out. If it works out I could even try to make it available to others (although I don't think I want to pay bandwidth costs, so I'd need to figure out a place to put it that wouldn't charge me for bandwidth).
Of course, all this is what I need to do before I even bother developing a portable app for reading the information. I'm not even going to think about that until I've done all this data processing. That part is probably more fun anyway. I'm interested in probability and statistics. I got a minor in math in college, but I somehow never went through a statistics course. I've gotten a few weeks into an online MIT course in the subject, but I've been putting off working on that while I've been doing this encyclopedia stuff. I'll get back to it though. It seems like it'll come in useful for data mining in any case.
Showing posts with label discussion. Show all posts
Showing posts with label discussion. Show all posts
Monday, June 29, 2015
Sunday, June 14, 2015
Read Britannica Books on Archive.org
I currently own the 2007 Encyclopaedia Britannica 15th edition as well as a reprint of the 1771 first edition. However, libraries won't let you check out an encyclopedia and few have old versions anyway. So here I'll try to link to all scanned versions of the Encyclopaedia Britannica on archive.org that I'm able to find.
If I missed any, someone please let me know!
Wikipedia has a summary of all the editions ever produced.
1797 (3rd edition) (18 volumes)
- Vol I (missing) or Vol I
- Vol II (missing) or Vol II
- Vol III or Vol III
- Vol IV
- Vol V
- Vol VI or Vol VI
- Vol VII or Vol VII
- Vol VIII
- Vol IX or Vol IX
- Vol X or Vol X or Vol X
- Vol XI
- Vol XII
- Vol XIII
- Vol XIV
- Vol XV (missing) or Vol XV
- Vol XVI
- Vol XVII
- Vol XVIII
1798 (3rd edition) (18 volumes)
1801 (supplement to 3rd) (2 volumes)
1803 (supplement to the 3rd) (2 volumes?)
1810 (4th edition) (20 volumes)
- Vol I
- Vol II
- Vol III
- Vol IV or Vol IV
- Vol V
- Vol VI
- Vol VII
- Vol VIII
- Vol IX
- Vol X
- Vol XI
- Vol XII
- Vol XIII
- Vol XIV
- Vol XV (missing)
- Vol XVI
- Vol XVII
- Vol XVIII
- Vol XIX
- Vol XX
1823 (6th edition) (20 volumes)
- Vol I
- Vol II (missing)
- Vol III
- Vol IV
- Vol V
- Vol VI
- Vol VII (missing)
- Vol VIII (missing)
- Vol IX
- Vol X
- Vol XI or Vol XI
- Vol XII (date is missing from title page. Seems to be a misprint)
- Vol XIII
- Vol XIV
- Vol XV
- Vol XVI (scanner left their fingers in the picture! :) )
- Vol XVII (missing)
- Vol XVIII
- Vol XIX
- Vol XX (missing)
1824 (supplement)
1888 (9th edition) (25 volumes)
- Vol I
- Vol II (missing)
- Vol III
- Vol IV (missing)
- Vol V (missing)
- Vol VI
- Vol VII
- Vol VIII
- Vol IX (missing)
- Vol X (missing)
- Vol XI
- Vol XII (missing)
- Vol XIII
- Vol XIV (missing)
- Vol XV (missing)
- Vol XVI
- Vol XVII
- Vol XVIII
- Vol XIX (missing)
- Vol XX (missing)
- Vol XXI
- Vol XXII
- Vol XXIII (missing)
- Vol XXIV (missing)
- Vol XXV (missing)
1890 (9th edition) (25 volumes)
1907 (The New Werner edition?)
1910 (11th edition) (29 volumes)
- Vol I or Vol I or Vol I
- Vol II or Vol II or Vol II
- Vol III
- Vol IV or Vol IV or Vol IV
- Vol V or Vol V or Vol V
- Vol VI
- Vol VII
- Vol VIII
- Vol IX or Vol IX
- Vol X or Vol X or Vol X
- Vol XI or Vol XI or Vol XI
- Vol XII or Vol XII
- Vol XIII or Vol XIII
- Vol XIV or Vol XIV or Vol XIV or Vol XIV
- Vol XV through XXIX are missing (all are in 1911?)
1911 (11th edition) (29 volumes)
- Full text of 11th edition
- Vol I through XIV are missing (all are in 1910?)
- Vol XV or Vol XV or Vol XV
- Vol XVI or Vol XVI
- Vol XVII
- Vol XVIII
- Vol XIX or Vol XIX or Vol XIX
- Vol XX or Vol XX or Vol XX
- Vol XXI or Vol XXI or Vol XXI or Vol XXI
- Vol XXII or Vol XXII or Vol XXII
- Vol XXIII or Vol XXIII or Vol XXIII
- Vol XXIV or Vol XXIV or Vol XXIV
- Vol XXV or Vol XXV or Vol XXV or Vol XXV
- Vol XXVI or Vol XXVI or Vol XXVI
- Vol XXVII or Vol XXVII
- Vol XXVIII or Vol XXVIII or Vol XXVIII
- Vol XXIX
Monday, May 25, 2015
Plain lists of facts are boring.
I thought about how I should format these posts. It would probably be boring to just list topics, yet many topics I just want to save without really commenting on them. So, the plan is to title the post after a particularly interesting topic. I'll comment on it, then in a MISC section, just list the rest of the topics I encountered that I just want to save for future reference.
I guess that means I can't make any posts unless I encounter an interesting topic?
Sunday, May 24, 2015
48 inches of concise knowledge
Last week I made a ridiculous purchase. I paid $700 for a full 2007 set of the Encyclopaedia Britannica. That's $700 for a 5 year old paper encyclopedia (encyclopaedia?).
Why? Am I a cranky old guy who doesn't get this new web thing? (Get off my lawn!) No, not really. At least, I'm not quite that old.
Am I a bookophile who just loves paper books (wik-whatia?) and refuses to get with the modern world? Well, no. I love my kindle apps and use google whenever I need to find out important things like how they did the driverless scenes in Knight Rider (a guy dressed up as a seat).
So what's the deal? The deal is that the great editors over at Britannica have spent about 250 years making articles that are as concise as possible. The paper form motivated them to fit as much information in as little space as possible. Wikipedia is great, but good gravy, concise it is not.
After years of reading news sites every day, I feel like things are just too repetitive. The pool of knowledge that they explore is just too limited. This is time I now plan on spending reading the Encyclopaedia Britannica.
When people celebrate holidays or make movies about wars, they almost always choose from a fairly limited set of wars. Everyone loves the Hitler, Washington, Lincoln, and Rambo wars. The countless other wars throughout history are barely mentioned. Those wars were great, but after years of hearing about them, I find them dull. I want to know about all the others. I hope Britannica can help me out.
I want to read the Micropaedia. Britannica actually has 32 volumes, but only the first 12 make up the Micropaedia. Most of the rest is the Macropaedia which has a lot more depth. I'll certainly spend time learning more about things that I find in the Micropaedia, but I'll probably just look them up online for depth. I bought these books so I could get access to the years of editorial writing and rewriting that distilled all the world's knowledge down into about 12000 pages (about 1000 pages per book) of 9 point type.
I've been asked why I didn't just go to the library. I could have read them there for free! Well, perhaps. Except for one problem. Libraries don't care about encyclopedias anymore! I went to 4 libraries. The first had a 2013 World Book (a children's encyclopedia). The second had a 2003 Britannica set. The third had a 2005 Britannica set with volume 1 missing. The fourth had the 2010 set! (the last one ever printed) But, they had it in storage. It wasn't even out on the shelves despite the tremendous amount of space in that library. (They were probably keeping it as a collector's item.)
Furthermore, despite not bothering to keep up to date, they also won't allow me to check out any of the volumes. Seems contradictory that they find them unnecessary, but they also won't let you check them out.
That's ok. Owning a set just feels right anyway. If I want to write notes in the margins or highlight things then I'll be able to do that in my own copy.
Will I actually make it through all 12000 pages? Maybe. Maybe not. My interests in woodworking, metalworking, cement casting, origami, balloon twisting, diorama building, gardening, kayaking, snorkeling, sewing (mostly Halloween costumes), Pac-Man collecting, piano playing, and making android games have all had various durations. Some are ongoing. Others have fallen by the wayside. But, keeping a blog about this will help me retain more of what I read, so that's the plan. I'll write about whatever I find interesting enough to be worth remembering.
I'll have to be careful about leaving too much metadata lying around though. If I let slip which volumes are where, you'll be able to follow my locations by watching which words I've been reading about. If I start writing about Ceausescu, Nicolae (he was such a reckless bastard!), you'll know that I'm in the room containing volume 3 (Ceara-Deluc).
Why? Am I a cranky old guy who doesn't get this new web thing? (Get off my lawn!) No, not really. At least, I'm not quite that old.
Am I a bookophile who just loves paper books (wik-whatia?) and refuses to get with the modern world? Well, no. I love my kindle apps and use google whenever I need to find out important things like how they did the driverless scenes in Knight Rider (a guy dressed up as a seat).
So what's the deal? The deal is that the great editors over at Britannica have spent about 250 years making articles that are as concise as possible. The paper form motivated them to fit as much information in as little space as possible. Wikipedia is great, but good gravy, concise it is not.
After years of reading news sites every day, I feel like things are just too repetitive. The pool of knowledge that they explore is just too limited. This is time I now plan on spending reading the Encyclopaedia Britannica.
When people celebrate holidays or make movies about wars, they almost always choose from a fairly limited set of wars. Everyone loves the Hitler, Washington, Lincoln, and Rambo wars. The countless other wars throughout history are barely mentioned. Those wars were great, but after years of hearing about them, I find them dull. I want to know about all the others. I hope Britannica can help me out.
I want to read the Micropaedia. Britannica actually has 32 volumes, but only the first 12 make up the Micropaedia. Most of the rest is the Macropaedia which has a lot more depth. I'll certainly spend time learning more about things that I find in the Micropaedia, but I'll probably just look them up online for depth. I bought these books so I could get access to the years of editorial writing and rewriting that distilled all the world's knowledge down into about 12000 pages (about 1000 pages per book) of 9 point type.
I've been asked why I didn't just go to the library. I could have read them there for free! Well, perhaps. Except for one problem. Libraries don't care about encyclopedias anymore! I went to 4 libraries. The first had a 2013 World Book (a children's encyclopedia). The second had a 2003 Britannica set. The third had a 2005 Britannica set with volume 1 missing. The fourth had the 2010 set! (the last one ever printed) But, they had it in storage. It wasn't even out on the shelves despite the tremendous amount of space in that library. (They were probably keeping it as a collector's item.)
Furthermore, despite not bothering to keep up to date, they also won't allow me to check out any of the volumes. Seems contradictory that they find them unnecessary, but they also won't let you check them out.
That's ok. Owning a set just feels right anyway. If I want to write notes in the margins or highlight things then I'll be able to do that in my own copy.
Will I actually make it through all 12000 pages? Maybe. Maybe not. My interests in woodworking, metalworking, cement casting, origami, balloon twisting, diorama building, gardening, kayaking, snorkeling, sewing (mostly Halloween costumes), Pac-Man collecting, piano playing, and making android games have all had various durations. Some are ongoing. Others have fallen by the wayside. But, keeping a blog about this will help me retain more of what I read, so that's the plan. I'll write about whatever I find interesting enough to be worth remembering.
I'll have to be careful about leaving too much metadata lying around though. If I let slip which volumes are where, you'll be able to follow my locations by watching which words I've been reading about. If I start writing about Ceausescu, Nicolae (he was such a reckless bastard!), you'll know that I'm in the room containing volume 3 (Ceara-Deluc).
Subscribe to:
Posts (Atom)