Research methods
I think I’ve always had certain stereotypes in my mind about research. I imagine a cutting-edge workplace, maybe not using the newest gadgets because these things cost money, but at least using the newest ideas. I imagine staff of research institutions applying the scientific method to boost their own productivity, instead of taking for granted the way that things have always been done. Maybe those were the naive ideas of someone who had never actually worked in a research field.
At the medical research institute where I work one day a week, I recently spent an entire seven-hour day going down a list of patient names, searching them on the hospital database, deciding whether they met the criteria for a study, and typing them into a colour-coded spreadsheet. The process had maybe six discrete steps, and all of them were purely mechanical. In seven hours, I screened about two hundred and fifty patients. I was paid $12.50 an hour to do this. It cost my employer 35 cents for each patient that I screened, and these patients haven’t been visited, consented or included in any study. They’re still only names on a spreadsheet. I’ve been told that I learn and work quickly, but I know I do this task inefficiently, because I’m not a simple computer program. I get bored. I make mistakes. Heaven forbid, I get distracted and start reading the nurses’ notes for fun because I find them interesting.
In 7 hours, I imagine that someone slightly above my skill level could write a simple program to do the same task. They wouldn’t screen any patients in those 7 hours, but once the program was finished, they could use it forever, or at least until the task changed and the program had to be modified. I don’t know how much it would cost the organization to employ a programmer; maybe it would cost more than just having me do it. I don’t know whether allowing that program to access the confidential database would be an issue. But it seems inefficient to pay human brains to do work that they’re bad at, that computers would be better at, even if those human brains belong to undergrad students who need the money badly enough not to complain.
One of the criteria I looked at when screening patients was whether they did their dialysis at a clinic in my hometown. They have to be driving distance, because my supervisor has to drive around the city and pick up blood samples to bring to our lab. I crossed out 30 names without even looking them up because I could see at a glance that they were a nearby city an hour’s drive away. How hard would it be to coordinate with the hospital in that city? Have the bloodwork analyzed there and the results emailed over? Maybe it would be non-trivially hard; I don’t know. I didn’t ask my supervisor because it isn’t my job to make management decisions. But medical research benefits everyone. A study with more patients produces data that’s statistically more valid, even if those patients live an hour’s drive away.
The office where I work is filled with paper. Floor-to-ceiling shelves hold endless binders full of source documents. Every email has to be printed and filed in a binder. Even the nurses’ notes and patient charts are printed off the database. It’s a legal requirement. The result is that we have two copies of everything, one online and one on paper, consuming trees. Running a computer consumes fossil fuels, of course. I don’t know for sure which is more efficient, paper or digital, but I do know that both is inefficient. I did ask my supervisor about this, and apparently it’s because digital records could be lost or deleted. How much would it take to make them durable enough?
I guess that more than my supervisor, I see a future where software will do my job, where technology allows a study to be coordinated across the whole world, where digital storage will be reliable enough. But how long will it take for the laws and regulations to change? For people to change? I don’t know how many of my complaints are valid. Maybe this is the optimal way to do research, but it doesn’t feel like it. It feels like a papier-mâché of laws and habits and trial-and-error. It doesn’t feel planned.
“It doesn’t feel planned.”
This made me think of a noteworthy corollary: airport control towers.
In the United States air traffic control is heavily regulated, and as a consequence the technology used is straight out of the 1970s—moving about little scraps of paper. In Canada, on the other hand (apparently we’re more economically free than the US? http://www.heritage.org/index/ ) there is less regulation, and the entire process is computerized.
For those of you who watch Breaking Bad, the disaster at the end of Season 3 probably wouldn’t have happened if the US adopted a similar system.
I think that ‘planned’ is a fallacy; systems as complex as Hospitals and Airports resist central planning, allowing the troops with their feet on the ground to design their own solutions often leads to a better result. I’m reminded of the time I worked at Bell Mobility’s Engineering dept—the only rule I had to follow, expenses wise, was “You cost the company twice what you’re paid per hour—if hiring outside is more efficient, go ahead and do it.” The entire office was loose and deregulated, and gosh darn, did we ever get things done!
Distributed decision making will occasionally result in 100 different researchers using $12.50/hour temps when pooling their resources would give them a computer programmer for less, but the hours spent in coordination would be expensive. On the other hand, the inefficient paper/electronic organization you bring up feels like the hallmark of over-regulation.
To quote Robbie Hanson, “Coordination is hard!”
No; ATC is completely computerized en-route, and heavily computerized at control towers. Directions given by voice are being transitioned to directions given via texts. I’ve done work for NASA in that area. (FAA uses NASA to do ATC research for historical reasons, basically as a way not to have to fire a bunch of people after the moon landings.)
I have seen the strips of paper for tracking flight plans… I don’t remember if they still use those. I won’t swear that they don’t. But it wouldn’t make any sense, as all that information is in the computer.
It is not, however, autonomous. There are still humans in the loop, which is a terrible safety hazard. Every year, air traffic controllers make over 10,000 errors serious enough to be logged and investigated. Of course, one mistake made by a computer is much more horrible and newsworthy than 10,000 comparable mistakes made by humans.
As a kid, I used to kill time in the LaGuardia TRACON ferrying around the paper strips to the controllers (and stacking up the plastic slides in pyramids & whatnot); I’m not sure the last time I went in, but I’m fairly sure it was between 1998 and 2001 (9/11 closed the TRACON to visits by controllers’ kids).
That’s interesting if correct. I would have guessed that this is due to NASA’s existence as NACA. Are you sure this is that late an addition?
This recent video by Reason Magazine would disagree; it argues to let a private corporation control Air Traffic (as in Canada), instead of bowing to the Unions: http://www.youtube.com/watch?v=KaPvJlPnc6E
I’m just going by what Nick Gillespe is claiming, but I’ve found Reason to be honest in their reporting and generally critical in their thinking,
When I saw that episode, my first thought was that it would be extraordinarily unlikely in the US, no matter how badly ATC messed up. TCAS has turned mid-air collisions between airliners into an almost nonexistent type of accident.
After writing that I thought “Actually, it probably still would have happened, because it’s such a great plot element.” Rule of Cool.
Much like organisations used to have typing pools, I think it would make sense for organisations to have scripting pools. When an employee recognises that a task could be automated, they hand it off to someone who can do a good job of it, and implement it in line with appropriate standards / QA / change control, etc.
Many IT departments do this. It makes users happy for awhile, but you quickly end up with hundreds of bits of code scattered around the organization that no one can keep track of.
You obviously need a script to keep track of them for you.
A few thoughts:
I am continually amazed at the amount of data that should be put and manipulated in a proper database that’s mucked about in spreadsheets instead.
Gene Callahan at ThinkMarkets recently had a post on how health care is especially bad when it comes to technology.
Eric Falkenstein says there are a lot of “skilled” jobs that could be replaced by computers too.
I had a job at an actuarial firm once and it would frustrate the hell out of me to see people constantly trying to do joins and other relational operations using Excel formulae, not knowing that that’s what they were doing, spending ages and ages on such tasks, holding Excel training sessions on how to use functions like INDEX and MATCH, using a bunch of custom-made macros to give names to columns and fill down formulas, and still making tons of mistakes that wouldn’t even be possible with a SQL query.
They were bleeding time and money at an astronomical rate, but it didn’t matter because their competitors were just as bad.
Excel is mostly used as a single-table database system. It has the important virtue of being an improvement on nothing.
Working with a ‘proper’ database is a skill I still lack, which I would very much like to acquire. I was hoping this job would help, but instead I just enter everything into Excel. (To be fair, you can manipulate data a fair bit using Excel formulas. I just don’t know all of them.)
Try MS Access. It is pretty simple to use, and has a nice export to Excel feature. Once you “get” Access it should be easy to move on to more advanced DB tools. Access also has a better UI that should make data-input work easier.
Thanks.
I agree in some sense, but disagree in another. I am fast at Excel. I don’t need to use to mouse, look at the menus, or pause to find anything I’m looking for, because I’ve internalized the keyboard shortcuts and created quick macros for the things I need to do. People get a little flustered when they see me work in Excel because it looks like it’s magically doing stuff, but it just comes from lots and lots of repetition.
Contrast this with a proper database, where I need to figure out some way to load the data in, make sure my query accounts for every place there might be a null value, then make some change that might break my previous queries if I need to add or change a column or something. And then if I need to take a slice of data and present it, I have to load it into Excel anyway.
For very large datasets with fairly static requirements, I use a database as is proper. But for anything less than 100K rows, give me a spreadsheet any day.
I had a job at an actuarial firm once and it would frustrate the hell out of me to see people constantly having to spend ages and ages doing ‘join’ operations using Excel formulae, not knowing that that’s what they were doing, spending ages and ages on such tasks, holding Excel training sessions on how to use functions like INDEX and MATCH, using a bunch of custom-made macros just to give names to columns, and making mistakes that wouldn’t even be possible with a SQL query.
They were bleeding time and money at an astronomical rate, but it didn’t matter because their competitors were just as bad.
Swimmer963:
This seems to me like it might be a very prudent measure. As far as I know, emails are legally considered as written documents bearing no less official weight than the most formal correspondence on letterhead paper. At the same time, it’s very hard for people to keep this in mind and refrain from using email carelessly, with no more caution and forethought than casual workplace talk. Forcing them to duplicate every email in paper involves a large overhead, but this may well turn out to be cost-effective by preventing all sorts of liabilities potentially incurred by careless emails.
(nods) Sure, if I really can’t expect employees to treat email as the kind of thing that it actually is, just because it is that kind of thing, then the next best alternative is to force them to treat email as something else.
Though if I were really going to go down that road, perhaps I’d do better to require that all employees jog around the building every time they send an email. Same benefit, plus it wastes less paper, and it’s healthier.
TheOtherDave:
I don’t know how serious you are about this, but even ignoring the problem of weirdness signaling, I don’t think this would be anywhere as effective. People usually have the correct instinct to treat paper documents and correspondence as an inherently serious and solemn matter, and coupling every email with producing a paper document is thus likely to be much more effective than coupling it with some meaningless ritual.
I’d say the casual attitude towards business emails is one of the greatest examples of practical irrationality in today’s world. I find it fascinating how many smart people, including lawyers and senior managers, who can’t possibly be ignorant of the legal weight of business emails, still can’t resist the urge to use them as a medium for casual chit-chat, offhand remarks, and informal discussions, until lawsuits hit them as a result.
My deep sympathies. I work in law, rather than in medicine, but my experience has been that the people making strategic decisions (i.e., management) are not even trying to optimize for any plausible set of performance criteria. Rather, they are all, to the last (wo)man, hard-core satisficers.
The managers’ daily activity is directed toward making problems go away, with minimum risk, minimum change, minimum (managerial) effort, and minimum cost, in descending order of priority.
Otherwise smart and competent people who know me fairly well and who I have talked to about this vigorously insist that as soon as I have my business/mortgage/reputation/ass on the line, or, failing that, after the first time I get burned by an innovation that doesn’t work out, I will adopt the exact same stance. I find that hard to believe (in the sense of having a very low prior for it), but the evidence is all pointing in the same direction, and I can’t think of a good reason (other than the low prior) why it’s not credible.
I don’t know whether you and I are actually optimizers at any deep level, or if we just see the kind of inefficiency that spends $100 on a page worth of patient names as being, itself, a major problem.
Either way, I hope we both manage to find work environments that value efficiency.
If you are using a SQL database and the requirements are a straight forward function of the available patient data, it would not take a programmer much more time to write a database query than to understand the requirements. This should take far less than 7 hours.
could you write the program in your spare time and run the program while you’re there, while making it seem like you’re working?
Maybe in 6 months. Right now it would take me weeks just to teach myself how to write a program on that level.
Hire someone in India or Russia?
The medical field (especially research) realizes that informatics is extremely important, but there’s not much consensus on implementation. I just caught a talk last week on the topic, and the speaker estimated that there were well over 200 “adjectivehere informatics” terms he found in a quick Google search—people just take whatever they originally did and add ‘informatics’ to it. Thus, terms like ‘pediatric endocrinology informatics’ abound in the literature...
You should also know that medicine, healthcare, and medical research as fields especially tend to be complex webs of conservatism, legalities, and other nasty little problems that must be overcome for change to occur (tip: check the cynicality on OvercomingBias). I think many technical people realize the cross-disciplinary opportunities available, but they just run aground.
Here are some of my recollections about the costs associated with transitioning to a paperless office.
I was recently employed for a month and paid $13 an hour archiving documents for a medium-sized (~40 fulltime employees) office in a much larger company. The office was transitioning to paperless records, and the entire previous year’s worth of printouts had to be scanned. There were three other people on my team. We each had a commercial scanner that the company had purchased new. The scanned documents were stored on multiply redundant company servers that had to be purchased for this transition. Every person in the office received a second monitor. The internal IT staff spent months on the transition, and a number of highly paid executives had to spend a not-insignificant amount of time deciding on the configuration of the final system. Additionally, another IT firm was contracted to set up some large portions of the system. While I was still there, the servers crashed and went down for a day and the office mostly halted, being unable to continue much of their work without access to their digital files. I witnessed frustrated staff vent some anger about the new system occasionally and support for it was mild at best.
Going paperless requires a very large commitment of resources up front and can significantly negatively impact productivity if everything doesn’t go exactly as planned. And even if things do go well, at that.
My workplace does things in a similar way, scanning in documents by hand without ‘interpreting’ them in any way. (The result is helpful; you can go on your computer and look at a patient’s chart without having to physically go to their hospital campus; but it’s also unhelpful in that you can’t run a keyword search on anything in the charts, because they’re saved as images as opposed to more search-friendly formats.) It looks messy and inefficient to ME that they’re keeping both paper and digital records, but I’m sure the immediate cost of making a full transition would be enormous.
Still, I can’t imagine that offices in fifty years will still be using this half-and-half method. As technology advances, maybe the transition will get easier; parts of the transition process itself could be automated, with software automatically converting scanned images into searchable text files. Either way, I think the transition has to be made eventually. (But that’s a personal opinion.)
See my Australian Electoral Commission example. I can assure you that even a basic image scan is far easier to deal with than all the physical paper all the time. Particularly in 2011 rather than 1993.
Mine too—a bank.
We’ve launched a company-wide project to estimate the cost-benefit relationship of scanning all new documents vs. scanning all new + existing documents vs. continuing like now. Perhaps not surprisingly, scanning new documents but not old is the most cost-efficient. This obviously depends on how often one needs to retrieve the documents.
At my company, servers but not scanners exist, and many people already have two monitors.
Don’t worry about fossil fuels, if you see the future where software will do the jobs. I wouldn’t worry about FF even if not, but that’s another thing.
You are an optimist about energy sources, then. What energy source do you think will replace fossil fuels? How long do you think the transition take? How easy will it be? I guess I’m fairly optimistic that SOMETHING will replace fossil fuels eventually, when the demand soars enough and scarcity sets in, but I don’t know enough about alternative energy sources to say which ones will play a major role.
Calling it optimism is a bit strong. Much of the world is already transitioning to nuclear power for non-transportation uses, and there are no technical obstacles to converting virtually all power generation in the long run.
I did not know that. Can you cite this information? What percentage of the world’s energy actually comes from nuclear?
I’ve been unable to find any clear description of long-term trends, but Wikipedia’s article on the prospective nuclear renaissance seems to suggest that it exists mostly in rhetoric and long-range planning at the moment, and nuclear capacity seems to have failed to track recent increases in global power consumption (data only to 2006, though). More reactors are being built than are being closed, but not all that many more in comparison with existing capacity.
Wow, what a nightmare. In bioinformatics, scripting tasks analogous to that is a big part of the job description. You have to download and correlate all sorts of data; but instead of processing 250 patients, you need to process a million genes, or a hundred million short reads from a sequencer; and scripting is necessary.
Another big part of the job is continually patching the duct tape used to glue all the scripts together.
Assuming that there are some legal requirements that the records be stored securely, you could back them up digitally with Tarsnap for $0.30 per GB transfer, and $0.30 per GB per month of storage. If you loosen your security requirements and go with Amazon EBS, you can do it for exactly a third of that. I expect competitors to offer similar pricing, but those are the ones that I’m familiar with. Both of these are fairly low-level options, and would require someone who knows what they’re doing to set up. Of course, printing everything out and storing it also takes a lot of effort, but it’s an ongoing effort that everyone takes for granted, whereas getting electronic backup set up would be something new, and therefore painful.
In 1993 I worked at the Australian Electoral Commission for a while. They had a similar requirement pertaining to paper, particularly the forms people fill in to enrol or re-enrol at their current address. (Showing up to vote is compulsory, so the AEC makes it really easy to enrol. There’s even a “walks” team that just goes door to door getting people to fill in their enrolment form.) They scanned every paper form to magneto-optical cartridges (the most compact replaceable storage technology at the time) and kept the paper filed in boxes, never to be touched. Maybe once a week a paper form would need to be fished out of filing because it had escaped scanning.
That is: this was a completely solved problem twenty years ago.
It is also a new decision that someone would have to take responsibility for making, who would be blamed if something went wrong. No one has to stick their neck out to continue the old way.
“It doesn’t feel planned.”
Alright, exercise time: rather than picking out the individual things wrong with this scenario, can you describe (at a relatively high level) what your plan would be, if you were to plan this? From scratch, with today’s technology, what would be the optimal way to run this sort of study?
OK, I’ll give it a go.
1) Start with a list of all hemodialysis patients in your area (city, province, or country depending on how participants you want.) The information on them should all be held in one database. (At my workplace, it’s split between two databases, only one of which I am cleared to access.) Say there are 100,000 names on the list.
2) Use a simple program to search through those names, marking true/false for various selection criteria. Even if the program isn’t perfect, say you reduce the list to 1,000 names. A human employee could then check over this much shortened list.
3) Contact the patients who qualify. Right now, the research staff have to drive to the clinic or hospital and see the patients during their dialysis in order to ‘consent’ them, or ask them whether they want to participate in the study. I don’t know how much of this is based on the regulations for medical research, but maybe the nurses could hand out pamphlets and patients could check YES or NO if they were interested in being in the study and willing to be contacted by telephone. This wouldn’t always work, but it could be helpful.
4) The consented patients need to have baseline, 6 month, 12 month and 18 month blood draws. I probably spend 8 to 10 hours a month making colour-coded packages of blood tubes to give to the nurses. What if instead we could mail them a box of blood tubes and sheets of blank labels, and send the label with patient code numbers by email? Assuming each clinic has the facilities to print, they could print the labels themselves, and use another cheat sheet we send by email to connect each patient to their number code. (For confidentiality and other reasons, the tubes can’t be stored with the patient names.) This would mean a little extra work for the staff at each clinic, but assuming most nurses can read and follow directions as well as I (not even a nurse yet) can, it would save hugely on labour overall.
5) Right now we use couriers to move blood samples to the labs. I can’t think of any ways to improve the efficiency of this step, but they probably exist.
6) Once the samples are studied, it would be nice to receive the data as a file by email, rather than as paper lab printouts that have to be entered into our database by hand, increasing the risk that an over-tired student employee like me will make an error. Some of our results appear in the main patient database under monthly blood work; another simple program could retrieve these.
7) At this point, analyzing the data is done on the computer anyway, but following this process, it might be in a more optimal format.
I’ve tried to stay within the bounds of today’s technology. Obviously research will get easier if and when portable ‘labs-on-a-chip’ can be made on desktop fabricators on location, with the results uploaded automatically into our files.