Understanding Python module usage on GitHub with python-api-inspect
Sunday, July 14th, 2019 | Uncategorized | No Comments
A SciPy 2019 lightning talk by Chris Ostrouchov introduced me to his project python-api-inspect. This provides an API for querying Python package usage on GitHub (and hopefully soon other source control systems).
It includes very flexible SQL queries, for example this one which we used to find packages that use the sympy.physics.quantum package within sympy:
If you click on the link you can fill in your own project and package to search for.
The Scientific Method
Saturday, July 13th, 2019 | Uncategorized | No Comments
A lightning talk I gave at SciPy 2019 in Austin, Texas,
A common view of the scientific method is that it consists of:
- formulating a hypothesis, and then
- conducting experiments to falsify or confirm this hypothesis.
This view has a few problems:
- It suggests a black and white picture where hypotheses are either right or wrong.
- It paints a picture of science as a large body of disconnected statements.
- It presents a narrow view of science where experiments are of primary importance, and
- I don’t think it matches how science actually happens.
These problems are far from academic. Science encounters a significant amount of skepticism & societal views are often polarized. I don’t think the common view of the scientific method is helping. When people are bombarded with many short disconnected statements and not given any tools to connect or reason about these themselves, it’s not surprising that many of them become confused.
I think a more useful view of the scientific method is as building and characterizing models
Let’s take vaccines as an example. I’m fairly sure that the measles vaccine doesn’t cause autism, and if someone came to me with evidence that it did, I’d be fairly skeptical. This isn’t because I’ve done a lot of experiments of my own though, or read a lot of papers by experimenters I trust. It’s because I have a simple model in my head of what the measles vaccine is — an attenuated form of the measles virus. Thus it seems unlikely that the vaccine would do something the virus didn’t. On the other hand The Cutter Incident in the 1950s where Cutter Laboratories distributed vaccines accidentally containing live polio virus seems very plausible in terms of the model.
If we’re going to win over science skeptics, we’re going to have to explain our models. This is a harder kind of educating than stating facts & hoping to be believed & hoping that we were right in the first place — but no one said this was going to be easy.
Science is about model characterization. We build mental models and characterize them. We expand their consequences. We understand when they’re valid and when they’re not and how accurate they are. We simplify and improve them.
Models come in many shapes and sizes. They might be as simple as a drawing, or as complex as the standard model of quantum mechanics.
Experiments still have a unique value — their measurements are independent of our models. But how we decide which experiments to do and how we interpret the results is very model dependent.
As an added bonus, this model-based view looks Bayesian and solves the “all ravens are black” paradox.
But more importantly, a common set of models allows us to have a conversation — even if we disagree about the details. They allow us to come to shared conclusions and have constructive disagreements — to have a shared mindset. Models that are well characterized can be depended upon in their domain of validity. They allow us to have the “what if …” conversations that are the
basis of policy.
For many important issues we can’t perform experiments. We only have one planet and one life to live.
Where to from here?
Saturday, October 3rd, 2015 | Uncategorized | No Comments
Closing speech at the end of PyConZA 2015.
We’ve reached the end of another PyConZA and I’ve found myself wondering: Where to from here? Conferences generate good idea, but it’s so easy for daily life to intrude and for ideas to fade and eventually be lost.
We’ve heard about many good things and many bad things during the conference. I’m going to focus on the bad for a moment.
We’ve heard about imposter syndrome, about a need for more diversity, about Django’s flaws as a web framework, about Python’s lack of good concurrency solutions when data needs to be shared, about how much civic information is locked up in scanned PDFs, about how many scientists need to be taught coding, about the difficulty of importing CSV files, about cars being stolen in Johannesburg.
The world is full of things that need fixing.
Do we care enough to fix them?
Ten years ago I’d never coded Python professionally. I’d never been to a Python software conference, or even a user group meeting.
But, I got a bit lucky and took a job at which there were a few pretty good Python developers and some time to spend learning things.
I worked through the Python tutorial. All of it. Then a few years later I worked through all of it again. I read the Python Quick Reference. All of it. It wasn’t that quick.
I started work on a personal Python project. With a friend. I’m still working on it. At first it just read text files
into a database. Slowly, it grew a UI. And then DSLs and programmatically generated SQL queries with tens of joins. Then a tiny HTML rendering engine. It’s not finished. We haven’t even released version 1.0. I’m quietly proud of it.
I wrote some games. With friends. The first one was terrible. We knew nothing. But it was about chickens. The second was better. For the third we bit off more than we could chew. The fourth was pretty awesome. The fifth wasn’t too bad.
I changed jobs. I re-learned Java. I changed again and learned Javascript. I thought I was smart enough to handle
threading and tons of mutable state. I was wrong. I learned Twisted. I couldn’t figure out what deferreds did. I wrote my own deferred class. Then I threw it away.
I asked the PSF for money to port a library to Python 3. They said yes. The money was enough to pay for pizza. But it was exciting anyway.
We ported another library to Python 3. This one was harder. We fixed bugs in Python. That was hard too. Our patches were accepted. Slowly. Very slowly. In one case, it took three years.
Someone suggested I run PyConZA. I had no idea how little I knew about running conferences, so I said yes. I asked the PSF for permission. They didn’t know how little I knew either, so they said yes too. Luckily, I got guidance and support from people who did. None of them were developers. Somehow, it worked. I suspect mostly because everyone was so excited.
We got amazing international speakers, but the best talk was by a local developer who wasn’t convinced his talk would interest anyone.
I ran PyConZA three more times, because I wasn’t sure how to hand it over to others and I didn’t want it to not happen.
This is my Python journey so far.
All of you are at different places in your own journeys, and if I were to share some advice from mine, it might go as follows:
- Find people you can learn from
- … and make time to learn yourself
- Take the time to master the basics
- … so few people do
- Start a project
- … with a friend(s)
- Keep learning new things
- … even if they’re not Python
- Failure is not going to go away
- … keep building things anyway
- Don’t be scared to ask for money
- … or for support
- … even from people who aren’t developers
- Sometimes amazing things come from one clueless person saying, “How hard can it be?”
- Often success relies mostly on how excited other people are
- Stuff doesn’t stop being hard
- … so you’re going to have to care
- … although what you care about might surprise you.
Who can say where in this complicated journey I changed from novice, to apprentice, to developer, to senior
developer? Up close, it’s just a blur of coding and relationships. Of building and learning, and of success and
failure.
We are all unique imposters on our separate journeys — our paths not directly comparable — and often wisdom seems largely about shutting up when one doesn’t know the answer.
If all of the broken things are going to get fixed — from diversity to removing the GIL — it’s us that will have to fix them, and it seems unlikely that anyone is going to give us permission or declare us worthy.
Go build something.
So what is this roleplaying thing anyway?
Sunday, August 16th, 2015 | Roleplaying | No Comments
I ran a roleplaying module [1] for some friends from work and after initially neglecting to explaining what roleplay is, I wrote this:
Roleplaying is a form of collaborative storytelling — a group of people gathering to tell a story together. This broad definition covers quite a range of things — one can tell very different kinds of stories and collaborate in very different ways.
What I’m planning to run is called “tabletop roleplaying” [2]. The stories told centre around a group of characters (the protagonists in a movie). Each person playing is in charge of one of these main characters, except for one person who has no character and instead handles everything that isn’t one of the main characters (they are a bit like the director of a movie).
Tabletop roleplaying is a little like a radio drama — almost everything is done by narrating or speaking in character. You’ll be saying things you want your character to say and describing the actions you want your character to take. Light acting, such as putting on accents or changing tone of voice or changing posture, can be quite fun, but is by no means a requirement.
The “director”, also called the “storyteller” or “DM” [3], describes the situations the main characters find themselves in, decides on the consequences of their actions and takes on the role of minor supporting characters and antagonists (often villains, because they’re exciting). The storyteller is also an arbitrator and a facilitator and attempts to maintain consensus and suitable pacing of the story.
Often you’ll want to have your character attempt an action that might not succeed [4]. For example, they might want to shoot a villain, charm their way past a bouncer, pick a lock or run across a burning bridge. In some cases success will be more likely than others. A character who is good looking or persuasive might find charming the bouncer easy. A character who has never picked up a gun might find shooting the villain hard.
The set of rules used to determine success or failure is called “the system”. The rules might be as simple as “flip a coin” or they might take up a whole book [5]. Dice are a commonly used way of randomly determining results with high numbers typically indicating more successful outcomes and lower numbers less successful ones.
Since the real world is very very complicated, the rules usually model different aspects of it in varying degrees of detail and this often sets the tone of the story to some extent. For example, a system for telling stories about bank robbers in the 1920s might have very detailed rules on vault locks, while a system for telling fantasy stories will likely have special rules for elves and dwarves.
All systems have shortcomings, and when these are encountered it’s usually the storyteller’s job to apply common sense and tweak the outcome accordingly.
The system I’m planning to use is an extremely simplified version of Dungeons & Dragons, 3rd Edition. The full rules run to many books. I’m hoping to explain the few rules we’ll be using in 10-15 minutes.
The story I’m planning to run focuses on a down-on-their-luck rock band about to enter a battle of the bands contest. The twist is that it’s a fantasy setting so there are elves and dwarves and, of course, in the hands of suitably skilled musicians, music is literally magical.
Some practical considerations
Someone has to supply the table to sit around. This is the person hosting the game. Ke can be a player or the storyteller or even uninvolved [6]. Traditionally the host also supplies a kettle and tea or coffee.
Everyone needs to show up and sit at the table, preferably roughly at the same time. This is surprisingly hard.
In order to remain at the table for prolonged periods, one needs things to nibble on. Traditionally people who are not the host bring snacks and drinks of various kinds. Roleplayers seem to take a perverse delight in bringing the unhealthiest snacks they can find, but this is perhaps a tradition best improved on.
Remembering the details of the story can be tricky, so it’s often useful to scribble short notes for oneself. A pen and paper come in handy.
I’ll give each player a couple of pages of information about their character. These are called the “character sheet”. The primary role of the character sheet is to supply margins to scribble notes and doodles in (see above).
It’s likely that time will fly remarkably quickly. If there are six of us, each person will get on average less than ten minutes of “screen time” per hour and probably a lot less given that the storyteller usually uses more than their fair share and there are always distractions and side tracks like discussing the rules or office gossip [7]. If we run out of time, we can always continue the story another day if we’re excited enough.
Lastly, the point is to have fun and tell an interesting story [8].
Glossary
Host: Person who supplies the table, and usually warm beverages like tea and coffee.
Table: Thing one plays at.
Storyteller: The person managing the world the story takes place in, the consequences of players actions and playing the minor characters and antagonists.
Players: The people who are not the storyteller.
Player character: One of the protagonists of the story. Each player has their own player character to narrate.
NPC: Non-player character. All the characters in the story who are not player characters.
RPG: Roleplaying Game. Also rocket-propelled grenade.
System: The rules used to determine the outcomes of risky actions.
Dice: Things one rolls. Usually because one is required to do so by the rules, but often just for fun.
Fun: The point. :)
Footnotes
[1] The module was This is Vörpal Mace. If you’re keen to play it, you can download it from Locustforge.
[2] So called because it usually takes place around a table.
[3] “DM” stands for “Dungeon Master” and is a silly legacy term from the earliest tabletop roleplaying games which mostly focused on a group of heroes running around vast dungeons full of traps and monsters. The storyteller’s role was mostly to invent traps and monsters, hence the title.
[4] Because otherwise the story would be very boring. :)
[5] A whole book is far more common. :P
[6] Although letting six people invade your house for an evening for an activity you’re not involved in requires a special kind of friendship.
[7] One can avoid this time-divided-by-number-of-people limit by having multiple scenes running concurrently. This is a lot of fun, but hell on the storyteller. :)
[8] And it’s easy to lose track of this amongst all the details of playing your character, keeping track of what’s happening and figuring out the rules.
[9] This footnote is not related to anything.
Character Creation 3000W
Thursday, August 21st, 2014 | Roleplaying | No Comments
by Simon Cross, Mike Dewar and Adrianna Pińska
Your character creation skills have progressed far beyond writing
numbers on paper. Your characters have deftly crafted manerisms and
epic length backgrounds. They breathe emotion and seem more life-like
than many of your friends.
Yet, somehow, when you sit down at a table to play your beautiful
creations, things don’t quite work out.
Perhaps the story heads in an unexpected direction, leaving your
creation out of place and struggling to fit in? Or maybe they’re fun
to play initially but their actions begin to feel repetitive and
uninteresting?
If any of this sounds familiar, read on.
Reacting to failure
It’s easy to spend all your time imagining a character’s successes —
their victories and their crowning moments — but what happens when
they fail? How do they respond to minor setbacks? And big ones?
Maybe they’re stoic about it? Perhaps it’s likely to cause a crisis of
faith? Maybe they react by doubling down and uping the stakes? Maybe
they see failure as an opportunity to learn and grow? Perhaps they’re
accustomed to failure? Perhaps they see failure as a sign that they’re
challenging themselves and pushing their abilities?
The dice and the DM are going to screw you. Make sure you have a plan
for how to roleplay your character when they do.
Philosophy
A character’s goals are things strongly tied to specific events. A
philosophy colours every situation. The two are often aligned, but a
philosophy is more broadly useful. It gives you a handle on how your
character might behave in circumstances where it is not otherwise
obvious what they would do.
To take a hackneyed example: your backstory might involve punishing an
old partner who screwed you. This goal could feed a number of
rather different philosophies:
- “I always keep my word, and I promised Jimmy I’d get him back.”
- “Any situation can be solved with enough violence.”
- “Karma controls the universe. What goes around comes around.”
The goal is the same, but each philosophy implies very different
day-to-day behaviour.
There are going to be times when other characters’ plots and goals are
centre-stage, and it behooves us as roleplayers to have a plan for
these awkward (and hopefully brief) moments. A philosophy allows your
character to participate in others’ plots as a unique and distinct
individual, rather than as a bored bystander.
Your character’s philosophy becomes vitally important when paradigm
shifts occur in-game. Setting changes erode the importance of lesser
goals and past history and create a strong need for a philosophy that
guides your character’s immediate responses and further development.
It may be interesting to construct characters with goals that
contradict their philosophy. For example, a pacifist might wish to
exact revenge on the person who killed their brother. This creates an
interesting conflict that will need to be resolved.
Randomly fucking with people is not a philosophy.
Interacting with colleagues
Your character is going to spend a lot of time interacting with their
colleagues — the other player characters — so it’s worthwhile
thinking about how they do that.
It’s tempting (and a bit lazy) to think of characters as relating to
everyone else the same way. This leads to loners and overly friendly
Energizer bunnies, both of which get old very quickly.
Avoid homogenous party dynamics.
If your character’s interactions with the other player characters are
all the same, you have failed.
Varied interactions also help make party disagreements more
interesting. Without varied interactions, you have to resolve all
disagreements by beating each other over the head with the logic stick
until consensus (or boredom) is reached. Unique relationships and
loose factions make disagreements more interesting to roleplay and
help the party find plausible lines along which to unite for a given
scenario.
If your character is part of a command structure, spend some time
thinking about how they respond to orders they disagree with. Remember
that the orders are likely issued by someone your character knows and
has an existing relationship with. What is that relationship?
Also keep in mind that your character has likely been given such
orders before, and since they appear to still be part of the command
structure, they’ve probably come to terms with this in some way that
both they and their immediate superiors can live with.
Obviously everyone has their limits, though — where are your
character’s? How much does it take for other player characters or
NPCs to cross the line?
Development
Sometimes even if you do everything right you find yourself in a
situation where your character is no longer fun to play. Maybe the
campaign took an unexpected turn or you’ve just run out of ideas for
them as they are. It’s time for your character to change — to embark
on a new personal story arc.
Great characters aren’t static. They grow and react to events around
them. Perhaps a crushing defeat has made them re-consider their
philosophy — or made them more committed to it? Or maybe frustration
with their current situation has made them reconsider their options?
It helps to think broadly about how your character might develop while
you’re creating them. Make sure you’d still find the character
interesting to play even if their stance on some important issues
shifted. Don’t become too invested in your character remaining as they
are. Be flexible — don’t have only one plan for character
development.
Your character’s philosophy and general outlook can be one of the most
interesting things to tweak. Small changes can often have big
ramifications for how they interact with others.
Don’t feel you have to leave character development for later in the
campaign! The start of a campaign is often when character changes are
most needed to make a character work well and it sets the stage for
further character development later on.
Sharing
Think about how you convey who your character is to the other
players. They’re probably not going to get to read your epic
backstory, so they’re going to have to learn about who your character
is in other ways.
Likely the first thing people will hear about your character is his or
her name — so make it a good one. It’s going to be repeated a lot so
make sure it conveys something about who your character is. If they’re
an Italian mobster, make sure their name sounds like they’re an
Italian mobster. That way whenever the DM or another player says your
character’s name, it reminds everyone who your character is.
The second thing people hear will probably be a description of your
character. Take some time to write one. Don’t rely on dry statistics
and descriptions. Stick to what people would see and remember about
your character if they met him or her for a few minutes. Don’t mention
hair colour unless hair is an important feature.
After introductions are done, you probably won’t get another
invitation to monologue about your character. So do it in character
instead. Tell the NPC about that time in ‘Nam. Regale the party with
tales from your epic backstory. As in real life, try not to ramble on,
but equally, don’t shy away from putting your character in the
spotlight for a few moments. Continually remind the others at the
table who your character is.
Last but not least, remember that the most epic backstory is pointless
if no one finds out about it. The point of dark secrets is for them to
be uncovered and for your character to confront them.
Epilogue
Don’t fear failure. Have a philosophy. Have varied interactions with
others. Embrace change. Share who you are.
Thanks
- Kululaa dot COMMMM!
- Mefridus von Utrecht (for a philosophy that involves others)
- Attelat Vool (for starting life after failure)
This article was also published in the CLAWmarks 2014 Dragonfire edition.
Using a Telkom Huawei modem under Ubuntu
Sunday, July 8th, 2012 | Uncategorized | No Comments
Telkom sell a wireless land-line phone build on the Huawei ETS6630 chipset. Under the hood it’s essentially a cellphone in a land-line form factor and it has a USB connector on the side that allows it to function as a USB GSM modem. However, there are a couple of obstacles to getting it running under Linux (specifically Precise):
- When first plugged into a USB port, it’s just a USB storage device (containing the Windows drivers). It needs to be sent a special command to make its GSM modem available.
- There is a utility for doing this (usb_modeswitch) but the version in Precise doesn’t pick up Telkom’s phone by default.
To help usb_modeswitch along, create /etc/usb_modeswitch.d/12d1:1011 containing the configuration needed for the Telkom phone:
# /etc/usb_modeswitch.d/12d1:1011 # Huawei ETS6630 TargetClass=0xff HuaweiMode=1
The “12d1″ and “1011” are the vendor and product code that identify the device. The TargetClass tells usb_modeswitch how to recognize when the GSM modem has been activated. HuaweiMode specifies the command to send to activate the device.
Then edit /lib/udev/rules.d/40-usb_modeswitch.rules to add:
# /lib/udev/rules.d/40-usb_modeswitch.rules
# Huawei ETS6630
ATTRS{idVendor}=="12d1", ATTRS{idProduct}=="1011", RUN+="usb_modeswitch '%b/%k'"
This tells udev to call usb_modeswitch when the Telkom phone is connected. Unfortunately these rules can’t just be put into /etc/udev/rules.d/ because 40-usb_modeswitch.rules includes additional actions needed to help recognize the device as a GSM modem.
If everything is working, you should now be able to restart NetworkManager and add a new Mobile Broadband connection via NetworkManager’s GUI. If things go wrong you can watch what is happening by tailing /var/log/syslog and watching what happens when you plug the Telkom phone into the USB port. Using “lsusb” and “lsusb -v” to look at what interfaces the Telkom phone is currently providing and poking the phone directly with “usb_modeswitch -v 12d1 -p 1011 -H” can also help. Note that running “usb_modeswitch” from the command-line won’t be enough to get the phone picked up as a USB serial device (that’s what all the additional rules in 40-usb_modeswitch.rules are for).
Hopefully I can get these fed back to the maintainer of the usb-modeswitch-data package and this blog post will become irrelevant. :)
Addendum: You might need to `modprobe options` and `modprobe usbserial` to get this to work (it looks like usb_modeswitch is meant to do this but my hacky additions don’t appear to get those steps to happen).
Apocalypse Pony Confronts the Carebears
Monday, January 18th, 2010 | Photos | No Comments
[Given to Anna and Ollie at their wedding.]
Drow Casts Fireball Before Leaving
Monday, January 18th, 2010 | Photos | No Comments
My mom gave me a sand art kit for my birthday. Result 1/3.
[Given to confluence on the occasion of her 28th birthday.]