Amersfoort (NL) python meetup¶

An introduction to web scraping - William Lacerda¶

William works at deliverect, the host of the meeting. Webscraping means extracting data from a website and parsing it into a more useful format. Like translating a list of restaurants on a

There’s a difference with web crawling: that is following links and trying to download all the pages on a website.

Important: robots.txt. As a crawler or scraper you’re supposed to read it as it tells you which user agents are allowed and which areas of the website are off-limits (or not useful).

Another useful file that is often available: /sitemap.xml. A list of URLs in the site that the site thinks are useful for scraping or crawling.

A handy trick: looking at the network tab when browsing the website. Are there any internal APIs that the javascript frontend uses to populate the page? Sometimes they are blocked from easy scraping or they’re difficult to access due to creative headers or authentication or cookies or session IDs.

A tip: beautifulsoup, a python library for extracting neat, structured content from an otherwise messy html page.

selenium is an alternative as it behaves much more like a regular webbrowser. So you can “click” a “next” button a couple of times in order to get a full list of items. Because selenium behaves like a real webbrowser, things like cookies and IDs in query parameters and headers just work. That makes it easier to work around many kinds of basic protection.

MicroPython - Wouter van Ooijen¶

A microcontroller is a combination of cpu, memory and some interfaces to external ports. https://micropython.org is a version of python for such low-power devices.

He demoed python’s prompt running on a raspberrypi micro connected via microUSB. And of course the mandatory lets-blink-the-onboard-LED programs. And then some other demoes with more leds and servos. Nice.

A big advantage of micropython is that it doesn’t care what processor you have. With C/C++ you specifically have to compile for the right kind of processor. With micropython you can just run your code anywhere.

You can use micropython in three ways:

• As .py sources, uploaded to the microcontroller.

• As pre-compiled .mpy code, also uploaded.

• As frozen .mpy included in the images

He showed a couple of possible target microcontrollers. A note to myself about the ESP8266: limited support, use .mpy. I think I have a few of those at home for should-test-it-at-some-time :-) Some examples: Pi RP2040, ESP32, Teensy 4.1.

A problem: RAM is scarce in such chips and python is hungry… You can do some tricks like on-demand loading. Watch out when using an LCD graphic display, that takes 150kb easily.

You have to watch out for the timing requirements of what you want to do. Steering a servo is fine, but “neopixel” leds for instance needs a higher frequency of signals than micropython is capable of on such a microcontroller. If you use a C library for it, it works (he showed a demo).

GraphQL in python? meet strawberry - Erik Wrede¶

Erik works as maintainer on the Graphene and the strawberry-GraphQL projects.

Graphql is a query language for APIs. It is an alternative to the well-known REST method. With REST you often have to do multiple requests to get all the data you have. And the answers will often give more information than you actually need.

With graphql, you always start with a graphql schema. You can compare it a bit to an openapi document. The graphql schema specifies what you can request (“a Meetup has a name, description, list of talks, etc”).

An actual query specifies what you want to get back as response. You can omit fields from the schema that you don’t need. If you don’t need “description”, you leave it out. If you want to dive deeper into certain objects, you specify their fields.

Strawberry is a graphql framework. It has integrations for django, sqlalchemy, pydantic and more. The schemas are defined with classes annotated with @strawberry.type and fields with python type hints. (It looked neat!)

He showed a live demo, including the browser-based query interface bundled with graphql.

Note: strawberry is the more modern project (type hints and so) and will later have all the functionality of graphene. So if strawberry’s functionality is enough, you should use that one.