Rotterdam python meetup¶

Microservices with Python for AI in radiology - Coert Metz¶

In radiology, people take a long time to become experienced. Medical school, MD, certified radiologist… And when they’re 68 they’re off to a pension. What they did at Quantib was to try and “scale radiology experience with AI”.

Detection and classification of prostate lesions. Same with breast MRIs. Brain shrinkage. They hope it increases the amount of MRI scans that can be processed. And also the quality of the analysis.

He demoed the application. There’s detection of brain regions in the software, for instance. When you compare two MRI scans at different points in time, you can see the difference and compare that difference with what you would see in a healthy person.

Hospital practice often means downloading radiology RMI images from a central hospital image storage server (“PACS”), taking them to a separate workstation for analysis and then going back with reports. This takes time, so it is sometimes omitted due to time pressure…

What they’re working on now is to run their AI software on a server and connect it to the image storage service. They designed their software as a bunch of microservices. Storage service, import, dispatch, workflow service, processing.

Nice idea: you can add exporter plugins to the system by means of docker containers.

Why microservices?

• Better scalable. AI on GPU nodes can be expensive. So it is more cost effective to only have to scale those AI services there and use regular nodes for the rest.

• Cloud-ready.

• It is easier to reason about a separate service in isolation. Failure modes and security is easier to figure out. And, important for a hospital, regulatory requirements are better manageable: risk management, cybersecurity.

• Of course, testing in isolation is easier.

Microservices are a bit harder to setup than a monolith. Especially when a large part of the team isn’t really experienced with devops type of work.

The core services and the front end are down with python and django. The services also mostly use django restframework. All the communication between the services is done with REST APIs. Extensions also talk to the APIs. Django restframework is mostly straightforward to use.

When designing an API, make it a nice clean clear consistent REST API. Follow REST good practices. Plural nouns (workflow/workflows). Use HTTP verbs (get/put/post/delete). If resources are nested, also nest them in the URLs. A puzzle: using the right HTTP status codes. There are nice decision trees available for that online. Don’t compromise!

The front-end consists of a separate django app that communicates with the back-end microservices. The user interface is done in javascript.

Testing: regular unittests plus django’s test cases. Javascript: jest (they have about 90% coverage). For integration testing they use PACT-python (consumer driven contracts). It is all done automatically on the continuous integration server. Getting the integration tests to work well was a challenge, btw. What helped was to return only minimal responses when mocking.

Deployment: docker in swarm mode (they’ll move to kubernates later). Docker secrets. Gunicorn+nginx. TLS everywhere: both ways between services. Regular single-way between the browser and the front-end service.

Home Automation and python - Bas Nijholt¶

Bas likes using programming in his life. For instance home automation: https://github.com/basnijholt/home-assistant-config

He didn’t care about home automation until he found a way to do it in python (home assistant) and he had a good use case. The use case was the elaborate video/audio system of a family member where they were moving in. It should not take six different buttons to finally get the TV running. Time to automate it.

Home automation is an expensive and time consuming hobby (“if it doesn’t cost time and if it doesn’t cost money, it is no hobby”). Changing lights. Turning heating on or off. When you go to the bathroom at night after you’ve been sleeping, don’t turn on the bright light in the toilet, but use a soft red light. Controlling the robot vacuum cleaner to only do its work when everyone is out of the house. Using a smart current meter connected to the washing machine that sends a message to your phone when it is ready. A packet sniffer between the regular thermostat and the heater to intercept and control it. A humidity sensor in the bathroom to detect when you’re showering: then the lights should stay on despite there being almost no movement :-)

Home automation should be fun and (mostly) useful. It should not invade your privacy or complicate your life.

Regarding complication, two things to keep in mind from the python philosophy:

• If the implementation is hard to explain, it is a bad idea.

• If the implementation is easy to explain, it may be a good idea.

So: home assistant. The big problem that it solves is that it ties everything together: all the various protocols (wifi, bluetooth, infrared, etc), all the various devices (temperature, humidity, switches, cameras, sockets, etc) and all the various companies… It is written in python. You have abstract “Devices” classes that can be subclassed. And there are lots of examples.

It is open source. Really open source, as it is in the top 10 github projects when you look at the number of contributors. There are lots of active developers. There are even four full time developers paid for by home assistant users!

He then showed his dashboard… A list of plants with their humidity level, for instance. Energy usage. Which lights were on or off. He sent his robot vacuum to a certain room through the web interface. He also showed a video he recorded: nice!

To start with, a raspberry pi and some sensors is enough. Probably you already have a few devices in home already that you can connect.

Detecting outages at scale - Sander van de Graaf¶

Sander works at down detector. A service that detects when something is down. They monitor loads of services (facebook, etc). Often they notice it earlier than the actual service itself.

They make most of their money from enterprise subscriptions that use it to monitor their own services and also the services they in turn depend on.

They’re using python and django and started in 2012. They initially used python-nltk to scrape twitter messages to determine if there was an outage for a certain service.

They started on physical servers (which he hates, as they tend to die sometimes), then moved to AWS and they’re now using serverless a lot. For serverless they switched parts from django to flask. Django is now used for database migrations and the admin, mostly.

Basically: async everything. A server creates jobs in redis, workers get jobs. A separate service monitors the queue size and increases and decreases the number of workers.

They use python RQ, “easy job queues for python”, which works with redis. He is really enthousiastic about it. It is really simple to use.

He then explained their setup, which uses loads of amazon services. A question from the audience was “don’t you have extreme lock-in this way?” His answer was: “if you use the cloud, go all-in”. If you can only use a small subset because you might want to move to a different cloud provider, you’re missing out on a lots of stuff. You ought to just use a regular virtual server, then. Much cheaper. If you have the money to use the cloud, go all in. Use all the nice tools and all the managed services.

What they also like: python’s @lru_cache cache decorator. Also: “black” for code formatting. Flask. Pipenv. https://codecov.io. statsd. Grafanacloud.

Personal Github projects - Ambar¶

He quicky showed some personal projects at https://github.com/ambardas .

Based on the book “deep work”, he wrote https://github.com/ambardas/make_my_day_planner to re-schedule his google calender a bit.

In between he showed how to use pytest, doctests and coverage. And github actions to automatically run it on github. Note: quite a lot of audience members mentioned that they like github actions, especially the speed.

Fun: https://github.com/ambardas/sorting_performance (currently, look in the development branch . A small project to determine the optimal on-a-table sorting process for supermarket footbal cards. You can optimize for speed or for you-can-do-it-while-doing-other-things.

See https://visualgo.net/bn/sorting for nice visualisations.