Worked Example- Using Urllib (Chapter 12)

Welcome to Python for Everybody. We are going to be talking about some code. If you want to download all the code, it’s right here. It’s all a single big ZIP file. And of all this sample code, the one I’m going to talk about is urllib1.py. It is not very exciting, it’s short. That’s what’s kind of nice about Python code. If we go and take a look at the code we played with just previously, which is Socket. The idea here is urllib is something that Python has produced for us to make Socket communications and HTTP communications a lot better. So this is making socket calls underneath it, but there’s a library that makes this quite simple. And so we have to do some imports, so instead of importing Socket, we’ll import these. And we are going to create a handle, urllib.request.urlopen and just pass in a string. So we’re not encoding this, we’re not sending get command. All this stuff we did in the previous Sockets example is gone. And then we can just put this as a for loop. And so we’re not using this lower level read and write code, we’re just using a for loop. And so that literally is going to read the text line by line. And the line does comeback as an array of bytes, so we have to do a decode. But then we got string, and then we can do strip on it. So this is like a super simple. Super simple. So there we go. Now the interesting thing is you also don’t see the headers, we just read the contents. Now it turns out in urllib, and we’ll see this in later more complex application. You can get the headers if you want. You can get various other things. So that’s urllib, a simple urllib tool. Now, we can also use this in urlwords to show you something quite interesting. And that is if you look at this from right here. Other than the decode, this is exactly the code we wrote to compute the words, right? So other than this line.decode, this is just a open something up. In this case, we’re going to open a URL. We’re going to create a dictionary. We’re going to loop through each of the lines in that thing. We’re going to decode them and then split them. So once you do line.decode, this is now a legitimate internal Python string. We split it, we run through the words and run the counts. And so this is exactly like code that we did before to run counts. And so python3 urlwords. And so that gives us a dictionary, which is the word frequency. And we could do all kinds of crazy stuff in here with the sorting and all the kinds of things. The important thing is, once you’ve done this and this, other than the need to decode these lines when you first get them. It really works, urllib makes URLs function inside Python very much like files. So these are short and to the point and very simple, and I hope that they were useful to you.