Artwork

Content provided by Gregory German and KALX 90.7FM - UC Berkeley. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Gregory German and KALX 90.7FM - UC Berkeley or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://player.fm/legal.
Player FM - Podcast App
Go offline with the Player FM app!

Bradley and Jessica Voytek

30:00
 
Share
 

Manage episode 309942940 series 3042656
Content provided by Gregory German and KALX 90.7FM - UC Berkeley. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Gregory German and KALX 90.7FM - UC Berkeley or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://player.fm/legal.

The Voyteks created the Brain Systems, Connections, Associations, and Network Relationships engine, or brainSCANr. The tool is used to explore the relationships between different terms in peer reviewed publications. http://brainscanr.com/


Transcript


Speaker 1: Spectrum's next


Speaker 2: [inaudible].


Speaker 1: Welcome to spectrum [00:00:30] the science and technology show on k a l x Berkeley, a biweekly 30 minute program bringing you interviews featuring bay area scientists and technologists as well as a calendar of local events and news.


Speaker 3: Good afternoon. I'm Rick Karnofsky. Brad swift and I are the hosts of today's show. We're speaking with Jessica and Bradley Vojtech. Jessica is a designer and a developer who earned her master's of information management and systems here at cal and has [00:01:00] worked on several UC Berkeley websites. She's also working on the future of science education through projects like ned the neuron. Brad is in NIH, N N I g m s postdoctoral fellow at ucs f. He got his phd from cal. He's a prolific blogger and Zombie expert. The void techs are here to talk about brain systems, connections and associations and network relationships or brain scanner. This website helps people explore [00:01:30] how neuroscience terms relate to one another in the peer reviewed literature. They've documented their project in our recent journal of neuroscience methods paper.


Speaker 4: Brad and Jessica, welcome to spectrum. Thank you for having us. Thanks. Can you tell us a little bit about brain scanner? Actually, I was at a conference here at cal hell by the CSA, so the cognitive science student association and Undergraduate Association here at cal that they had several neuroscientists and cognitive scientists come and give presentations and I was one of those people. I [00:02:00] was on a panel with a Stanford cognitive scientists at the end of the day. It was a Q and a. We got into a question about what can be known in the neurosciences and I had mentioned that the peer reviewed neuroscientific literature probably smarter than we are. There's something like 3 million peer reviewed neuroscientific publications and I was saying that that is just too many. There is no way for anybody to to integrate all of those facts and I said if there some automated or algorithmic way of doing that, we could probably find some neat stuff out and he disagreed with me pretty strongly on the panel [00:02:30] and I sort of stewed on that for awhile.


Speaker 4: That ended up becoming the brain scanner project actually, which is using text mining to look at how different topics in the neurosciences relate to one another. We had conversation about this and I had just started about six months before my a master's program at the School of Information. So all of the stuff that he was saying really jived with what I was learning. So we got together after that. We talked about it off and on sort of over dinner and stuff occasionally, but I think it was [00:03:00] right around won't. Right, right before we found out you were pregnant. Right, right around Christmas when we first actually sat down together to work on it and that was just a random evening. We didn't have, well, we didn't have a baby at the time, so we didn't have much else to do. Brad was working on this thing and he said, you know, I've been working on this all day.


Speaker 4: I'm trying to get this algorithm to work and see if we can get any results out of this. And I kind of challenged him. I said, I can do that faster than you can started taking my course. I had [00:03:30] all of these new skills that I just wanted to kind of show off. And I did. She actually beat me. You guys were both sort of where we were. We're basically coding. Yeah. We're sitting on the couch. Not really cause we weren't actually doing it together. We are using two different competitor competing. Exactly. So who do you see as the audience for brain scanner? Well I know the answer to that someone. Right. So I have colleagues who tell me a lot of Grad students actually mostly a who say that they use it as a stop for [00:04:00] searching. The peer reviewed neuroscientific literature. So pub med, which is the surface run by the National Library of medicine, which is part of the national institutes for health index is a lot of these peer reviewed biomedical journals.


Speaker 4: Their search engine is quite good but it returns just textual information. You know, you just, you see the 20 most recently published papers or you know, however you want to sort it related to the search term or of interest. Yeah. So basically anybody who wants to create an app can get access to this data. You have to follow certain [00:04:30] rules, but otherwise you can get the information out of this database easily. In a, in a sort of standard format, we provide a graphic or a visualization layer on top of the search so you can put in one of these search terms and you can see here are the topics that relate to it very strongly in literature. Statistically speaking, you know, uh, by that I mean here are the words or terms that show up a lot in papers with the term memory for example. We also then list the papers that are related and you can see the full list of terms and [00:05:00] how it relates to different topics and things like that. Um, if I want to look at a brain region and say, okay, what are the other brain regions that are related to this can really quickly visually see that based upon these 3 million publications that we, we searched through


Speaker 2: [inaudible]


Speaker 4: you are listening to spectrum on k a l x Berkeley. We're talking to with user interface developer, Jessica Vojtech. And neuroscientists, Bradley Wojciech about brain scanner


Speaker 2: [inaudible]


Speaker 4: [00:05:30] do you see other potentially valuable ways you can harness PubMed's data or other reference sources? Yeah, absolutely. So one of the aspects actually of the paper that we published was ways to address that, that very question. Uh, initially we tried to publish the paper just as a here's a, here's a resource or one of the editor's version on that rejected the paper, said, you know, what, what can you do with this? And a, of course, you know, this is something we've been thinking about. And [00:06:00] so I tried to build a proof of concept. So one of the, one of the things that we showed statistically speaking that you can do with this, does the data they call hypothesis generation or semiautomated hypothesis generation. And this works off of a very simple idea. Um, it's almost like recommend their algorithms and um, like linkedin or Facebook or something like that.


Speaker 4: You know, it's like if you know this person, you might know this person, kind of a friend of a friend should be a friend idea. You know, Rick and I, I know you and you know Rick, maybe you have a friend named Jim. And so statistically speaking, [00:06:30] Jim and I might get along right because you and I get along and you, and he'd get along, especially if I and Jim get along. And so you can go through algorithmically and say, you know, in the literature if Migraine for example, which is the example you give in the paper, uh, is strongly related to a neurotransmitter Serotonin, which I didn't know before we made the website actually, um, that in the medical literature there's a whole serotonin hypothesis from migraines I guess because Migraines respond to, uh, antidepressants, which are usually serotonergic drugs. So anyway, Serotonin [00:07:00] and Migraines are very strongly related and neuroscientists know a lot about the basic physiology of Serotonin, where in the brain is expressed and things like that.


Speaker 4: And so on the neuroscience side, we know that Serotonin is very strongly expressed in, in a brain region called the striatum, which is sort of deep frontal part of the brain. And, uh, there's thousands of papers that talk about Serotonin and Migraines and Serotonin in this brain region, the striatum respectively, but there's only 19 papers or something like that to talk about that brain region and migraines. [00:07:30] And so statistically speaking, maybe we're missing something here, right? Maybe just nobody's really looked at this connection between migraines in that brain region. Maybe there aren't papers published on it because people have looked and there's nothing there. But, uh, that's why it's somewhat automated. You can go through this list of recommended hypotheses and you as an expert, I can go through that list and say, oh, some of these are nonsense. Or Oh, that's, that could be interesting.


Speaker 4: Maybe. Maybe we should look into that. So it gives you a low hanging fruit basically. Yeah. And so that would be something [00:08:00] eventually I would like to build into the site. Are you continuing to analyze new papers as they enter in pub med? We haven't rerun it for awhile. I think there's something on the order of 10,000 new papers published every month in the neurosciences. But when you're standing in the face of 3 million, it's sort of drop in the bucket. So we, we worry running it every month or two. Um, but the results really don't change very quickly. Right. It's pretty stable. So yes, we, we should actually [00:08:30] run it again. It's been about six months or so. If you guys actually like, well I mean as a or perhaps how, you know, the ideas in the literature might change. For instance, that's actually something that I did do.


Speaker 4: Um, I eliminated the searches to just bring the regions, so how different brain regions relate to each other across time. So I did a search for all papers published up to like 1905, which wasn't very many. Of course not in your, you know, you have an exponential increase in the number of being published. Okay. But then again, I ran it again for all papers published to like up to 1935, [00:09:00] 55, 75, 95 and you know, 2005, right? Uh, or 2011. And you could actually see how our understanding of how different brain regions relate change over time. And that was kind of neat. Um, if I was going to be a little bit statistically, uh, stronger about this, what I should have done in the original paper, and I didn't think about it until after we republished it was I should've run the semiautomated hypothesis generation algorithm, uh, on a limited amount of data. So I test data set up to like say 1990 [00:09:30] and then found plausible hypotheses from that Dataset and then run it again on the entire thing and see, you know, if we had found new things. And you know, if that corroborated what we've learned in the last 22 years.


Speaker 2: [inaudible]


Speaker 5: you're listening to spectrum on k a l x Berkeley. We're talking with Brad and Jessica Vojtech about brain scanner. They're a site to show links [00:10:00] that may exist between brain structures, cognitive functions, neurological disorders, and more as data mined from the academic literature.


Speaker 2: [inaudible]


Speaker 4: I mean this is a side side project for us. Yeah,


Speaker 1: it was two weeks in $11 and 50 cents. And what did that go to? Um, coffee and coffee. Yeah. [00:10:30] Um, no it, it went into the Google app engine server time. So we actually were able to use Google app engine to distribute the processing, which is also what made mind my code a little bit quicker to run through all of this data.


Speaker 4: I was doing single queries at a time and because we have 800 terms in the database and we have to do how every term relates to every other term, it's 800 squared,


Speaker 1: try to buy two essentially. And then there's the roundtrip between between his your machine and the um, [00:11:00] pub met database. So, you know, you're making requests, you're making requests, making requests anyway. It was maybe three days, three days or four days. And I was able to do it in about two hours by um, putting it into the cloud and using app engine. So that $11 and 50 cents went to paying for the service and agree to say a hundred squared divided by two minus 800 a lot. So do you want to talk about how that dictionary of keywords was generated?


Speaker 4: Initially I [00:11:30] had wanted to try and figure out how brain regions relate this. This grew out of my phd work actually at Berkeley. I worked with Professor Bob Knight who used to be the head of the neurosciences institute, Helen Wells neuroscience here. And my phd thesis was looking at how to brain regions, the prefrontal cortex and the Basal Ganglia related to working memory. And as I was standing for my qualifying exams, I was trying to figure out, okay, what are the brain regions that send inputs to [00:12:00] [inaudible], which is one of the parts of the Basal Ganglia and where he dies this ride in project two. And in order to figure that out, I spent, I don't know, two months off and on three months off and on over at the biomedical library here, digging through old, uh, anatomical papers from the 1970s and basically drawing little hand-drawn charts to try and figure out how these things connected.


Speaker 4: And it really surprised me. It was frustrating because you know, here we are in, well, when [00:12:30] I was doing this, it was like 2008 right? And all I wanted to know is how different brain regions connect. And I was like, why can't I just go to a website and say, okay, striatum, what are its inputs and outputs? Like we have that information, right? Why can't I do that? Um, and so anyway, that was one of the motivating factors for me also. And there's actually a paper published in 2002 called neuro names. And then this researcher was trying to create an ontology of, of brain region names Ryan. So the terms that we use now in 2012 aren't necessarily [00:13:00] the same that people were using back in 1900 when they were first describing the basic anatomy. And so you have some Latin names for brain regions.


Speaker 4: You have modern names for brain regions, you have names for different groupings of brain regions. So I referred earlier to the base like Ganglia, uh, and that is composed of, you know, maybe five different brain regions. And if I talk about three of those brain regions, uh, can I give examples? Is the putamen and the Globus Pallidus, uh, Globus Pallidus is actually contained [00:13:30] of two separate ones. And the putamen and Globus Pallidus if you combine them together or known by one name. But if you combine the putamen with the striatum, that's a different name. And so you actually have these weird venn diagram overlapping naming Schema.


Speaker 1: There's a significant vocabulary problem, which is the term that we use in the information sciences. Basically the fact that you have multiple names for the same thing and you have the same name for some different things. So you know this venn diagram idea. Um, so yeah, [00:14:00] if you're going to use a very simple search algorithm, you have, you can't do it, you wouldn't, you're not going to get all of the results. So, um, I think our system tries to solve that vocabulary problem a little bit.


Speaker 4: And then there's actually a researcher, um, Russ Poltrack drag, who used to be a faculty of neuroscience at UCLA and I think he's in University of Texas now. And he actually tried to create an ontology for cognitive term. So in cognitive science and psychology and cognitive neuroscience, you know, we have terms like working memory [00:14:30] and attention and in they're trying to create a whole ontology for how these different things really. So like working memory as part of memory, which you know, in memory also contains a longterm memory. And so we'll use his first attempt as a dictionary as well. And then we went to the NIH website and they've got a listing of all these different kinds of neurological disorders and we use that. So we pulled a bunch of publicly available data basically and use those dictionaries as our starting point.


Speaker 1: And then we [00:15:00] also took suggestions from the people on our website almost immediately we started getting requests for more and different terms. So you had the, when you find two keywords that appear in a paper together, you assume that they're actually related. Can you comment on if people might have demonstrated that they're not actually related, how does that affect your system? Like some, like an instance in which, uh, it says this brain region is not connected to this other [00:15:30] brain region, right? Um, yes, we have assumed that there's a publication bias that if there is not a connection then someone does not publish a paper about that.


Speaker 4: Okay. And negative publications or negative findings go very under reported in the scientific literature.


Speaker 1: Right. So we're hopefully taking advantage of that. Hopefully the law of large numbers means that our data is mostly correct and it does seem to be that way. The example that Brad gave, uh, with the Allen Brain Atlas, [00:16:00] that there is certain corroborating evidence that seems to suggest that this is a, at least plausible connections. There's obviously no one say that better. No, that's perfectly scientifically accurate. I tend to get a little bit specific when I'm talking about this kind of stuff.


Speaker 4: Is there already some sort of bias that might drive certain kinds of papers up? If the paper has a lot of buzzwords, perhaps it suddenly becomes more important. Do you 100% yes, absolutely. There are always [00:16:30] hot topics. Uh, and that shows up for sure


Speaker 1: only because there's more papers published on that subject. We don't currently have a any kind of waiting per paper.


Speaker 4: Yeah. Like when you go into the website and you'd do something like, um, there's a brain region called the Amygdala and you know, it'll be very strongly associated with fear. And so that's actually one of my concerns is problem getting these biases. So, you know, there's a lot of literature on this brain region, the Amygdala and how it relates to fear, but it certainly does a lot more than just processing fear, [00:17:00] right? It's this general emotional affective labeling sort of idea that anyway, that's, that's neuroscience specific stuff, you know, and brain region called the insula and disgust or love or you know, these other kinds of strong emotions. And so yeah, it definitely reflects certain biases as well. And we, we try and quantify that even to an extent a little bit. So again, using the Allen Brain Atlas data, we show from our Dataset, what are the top five brain regions that express or that are related to dopamine, for example.


Speaker 4: And in the real human brain, what are the top five brain [00:17:30] regions that express dopaminergic related genes? And you can actually see that there's a very clear bias. So one of the regions that expresses dobutamine very strongly is very hard to study. Neuroscientifically speaking. It's, it's deepen deep part of the brain. It's hard to get any, it's very small, so you can't get it from like brain scanning expresses a lot of dopamine, but people don't study it and we can actually quantify then some of these under-studied relationships, right? We're like, here's a brain region that we know expresses a lot of dopamine, but there's a a hundred papers only and another [00:18:00] brain region that's very sexy and too about domain has 10,000 papers. Right? So our system shows you an example well of the current state of scientific literature. So it's not necessarily 100% correct, but it reflects what scientists think as a whole at this point. Yeah, I agree. And we try and be very careful about that in the paper and in talking about it like we are right now


Speaker 2: [inaudible]


Speaker 5: [00:18:30] you are listening to spectrum on k a l x Berkeley. We're talking with user interface developer, Jessica Vojtech and neuroscientist Bradley Vojtech about brain scanner.


Speaker 2: [inaudible]


Speaker 4: I was really surprised you. I taught neuro anatomy for three semesters here at Berkeley and you know, so I know the anatomy pretty well. And on your first ran it, I had one of those like yes, kind [00:19:00] of moments like I can't believe this work because it really does find all of these clusters really nicely. And that was a very pleasant surprise because technically speaking it couldn't have been any other way. Like it just has to, you know, I mean these topics co-occur a lot, so it should be that way, but it's always nice to see something like that work. Brian, I wanted to ask about the journals that you sent the paper off too. How did you pick them? Art Of picking a journal where to send a paper. It's actually really hard. So certain journals get [00:19:30] more readership than others. And then there's the open access factor.


Speaker 4: So I'm, I'm a big open science, open data advocate and so I try and shoot for that. I had forgotten, there's actually sort of a, a very wide protest of Elsevier, which is one of the publishing companies right now. And the journal that published my papers and Elsevier Journal, but, uh, I had signed the petition and I was part of that Nash shortly thereafter. That would have impacted my decision had I been thinking about it. Yeah. And yeah, so it's mainly a balance between readership and expectation and you sort [00:20:00] of get a feel after publishing a few papers of what editors are looking for. And so yeah, I am the one that has experience with navigating the academic publishing environment. Yeah. So yeah, we sent it out to a lot of journals and, uh, mostly it didn't pass editorial review, which means that there's an editor that decides whether or not conceptually it will be interesting for their journal to publish it once got center review at a journal and they're like, well, it was sort of torn.


Speaker 4: There were four reviewers, four pure reviewers, [00:20:30] and two of them were fairly enthusiastic and the other two are like, this is cool, but so what? Right. Um, and the general consensus was it didn't fit with the theme of the Journal. The Journal of neuroscience methods point really well and your reviewers are very, and um, actually there's a figure at the end of the paper where we did some integration with the Allen Brain. Alice Paul Allen, one of the co founders of Microsoft who is a cuisine heir, has put half a billion dollars into this institute. [00:21:00] Initially the goal was to map, uh, the expression of all of these different genes in the human brain, in the mouse brain, and they made all that data publicly available. And so we use that as a test data set. So we said, okay, where are these different, uh, neurotransmitter related genes actually expressed in the brain and what does our system think about wearing the brain? These neurotransmitters are, there's a week but significant correlation between the two, which suggests that our system reflects actual reality to a certain extent at least. [00:21:30] And that was a suggestion I got from one of the peer reviewers and that was really good. It was a lot of extra work, but it ended up being a really good addition to the paper.


Speaker 1: But both of you guys are involved in science education and science outreach. So I was hoping you can comment on that. I'm actually starting a project with a friend of mine building a neuroscience kids books. So we're going to teach neuroscience to elementary school kids with an electronic ebook featuring the neuron. Yes, featured his name is ned the neuron. He's a pure middle cell and he works in the motor cortex of the brain. [00:22:00] And is the neuroscience focus partly driven by bad or do you have any sort of personal interest in as well? I do have a personal interest in and I, I, you know, obviously it's convenient that my husband is a neuroscientist, but actually the character and the original story idea is my partners who's also a neuroscientist and phd in neuroscience here at cal here at cal.


Speaker 4: Yeah. I get this question a fair amount. Like why do I do blogging and outreach and things like that. So there's actually a few answers to that. One I find blogging, uh, helps me [00:22:30] do better science. If I have to figure out a very simple way of explaining something, then I feel like I understand it better. It's sort of like one of the best ways to learn something is by trying to teach it. Right. I had a very strange path to academia. I actually got kicked out as an undergraduate from the university. I had to sort of beg my way back in because my grades were pretty low. You know, a couple of people help me out along the way and that were pretty important to me. And I think a lot of Grad students have this experience where they, they feel like they don't belong there in in sense that like, oh [00:23:00] my God, I'm not smart enough to do this.


Speaker 4: You know? And when I look at the resumes or cvs of, you know, tenured faculty here at Berkeley, right? It's just paper after paper and award and amazing achievement and you're just like struggling to even understand how to write a paper and it seems just like this daunting, intractable problem. And so because of that, I actually have a section in my CV where I actually list every time a paper has been rejected. I've actually had graduate students tell me that. That's been kind of Nice to see that you know, you see somebody who's doing pretty well and you see that, you know, in order to get there [00:23:30] you sort of have to slog through a lot of crap.


Speaker 1: Did you plan to work together some more? I think so. You know, we're obviously working together to raise a son right now. We actually were talking on the way over here about trying to implement some of the ideas we've


Speaker 4: been talking about that people have suggested. I think we could definitely do that. Yeah, there's definitely a lot of overlap. I'm very interested in dynamic data visualization and that's something that Jesse's is obviously getting quite quite good at and so I'd [00:24:00] like to start doing that for a lot of my research papers as well. Brad and Jess, thanks for joining us. Oh, thank you very much for having us. Thank you so much for having us.


Speaker 2: [inaudible]


Speaker 6: and now for some science news headlines. Here's Renee Rao and Brad sweet


Speaker 2: [inaudible]


Speaker 7: [00:24:30] the Berkeley new center reports researchers at the University of California Berkeley are gathering evidence this fall that the Feisty Fox squirrels scampering around campus or not just mindlessly foraging for food but engaging in a long term savings strategy to track the nut stashing activity. The student researchers are using GPS technology to record all of the food burials and in the process are creating [00:25:00] an elaborate map showing every campus tree building and garbage can. Miquel Delgado a doctorial student in psychology heads the squirrel research team in the laboratory of UC Berkeley, psychologist Luchea Jacobs. The research team is replicating the caching experiment on humans by timing students as they burry Easter eggs on campus and try to find them. We're using humans as a model for squirrel behavior to ask questions that we can't ask. Squirrels still got us said the group has a cow squirrels website to promote their work.


Speaker 6: [00:25:30] UC Berkeley professor of cell and molecular biology and chemistry. Carolyn Bertozzi has won the 2012 Heinrich Violin prize. Professor Bartow Z has founded the field of bio orthogonal chemistry. In her groundbreaking approach, she creatively exploits the benefits of synthetic chemistry to study the vital processes within living beings. Professor Dr Volk Gang Baumeister, chair of the board of Trustees of the Heinrich Violin Prize says of Professor Berto z. [00:26:00] Her breakthrough method to identify sugar patterns on the cell surface is a milestone for our understanding of the functions of sugars in health and disease and paves the way for novel diagnostic and therapeutic approaches.


Speaker 3: Irregular feature of spectrum is a calendar of some of the science and technology related events happening in the bay area over the next two weeks. Brad swift and Renee Rao join me for this. The second annual Bay Area Science Festival is wrapping up this weekend. [00:26:30] Highlights include art in science and gallery gala showing the intersection of image and research tonight at the Berkeley Arts Festival, Gallery Science superheros tonight at the Tech Museum in San Jose and discovery days at at and t park tomorrow November 3rd from 11:00 AM to 4:00 PM last year more than 21,000 people showed up to this free event this year. There are more than 150 exhibits. Visit Bay area science.org for more information about any of these [00:27:00] great activities and to see their regular calendar of science goings on.


Speaker 6: Big Ideas. Berkeley is an annual innovation contest that provides funding, support, and encouragement to interdisciplinary teams of UC undergraduate and graduate students who have big ideas. The pre-proposal entry deadline is 5:00 PM November six 2012 all pre-proposals must be submitted via the online application on the big ideas website. Remember there are big idea advisers to help students craft [00:27:30] their pre-proposals. You can drop in at room 100 Blum hall during scheduled hours or email advisers to schedule an appointment at another time. Check the big ideas website for advisor times or to make an appointment. There will also be an editing blitz November 5th from five to 8:00 PM in room, 100 of bloom hall advisors and past winters will be available to provide applicants with valuable last-minute insights and advice on your pre-proposal. This is a great opportunity to hone your proposal and get support from those [00:28:00] who know what it takes to build a successful big idea. The big ideas website is big ideas.berkeley.edu


Speaker 7: on November 8th the center for ethnographic research will hold a colloquium to understand cancer treatment trajectories using an array of ethnographic data. The Speaker Daniel Dohan and associate professor in the Phillip r Lee Institute for Health Policy Studies. We'll discuss this research about inequality and culture with a focus on cancer. He will focus on his most recent study which examines how patients [00:28:30] with advanced diseases find out about and decide whether to participate in clinical trials of new cancer drugs. The event, which is free and open to the public, will be held from four to 5:30 PM at 25 38 Channing way


Speaker 2: [inaudible]


Speaker 5: the music you [00:29:00] heard during say show was spend less on and David from his album book and Acoustic, it is released under a creative Commons license version 3.0 spectrum was recorded and edited by me, Rick Karnofsky, and by Brad Swift. Thank you for listening to spectrum. If you have comments about the show, please send them to us via email. Our email address is spectrum dot k a l x@yahoo.com [00:29:30] join us in two weeks at this same time. [inaudible].



Hosted on Acast. See acast.com/privacy for more information.

  continue reading

78 episodes

Artwork
iconShare
 
Manage episode 309942940 series 3042656
Content provided by Gregory German and KALX 90.7FM - UC Berkeley. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Gregory German and KALX 90.7FM - UC Berkeley or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://player.fm/legal.

The Voyteks created the Brain Systems, Connections, Associations, and Network Relationships engine, or brainSCANr. The tool is used to explore the relationships between different terms in peer reviewed publications. http://brainscanr.com/


Transcript


Speaker 1: Spectrum's next


Speaker 2: [inaudible].


Speaker 1: Welcome to spectrum [00:00:30] the science and technology show on k a l x Berkeley, a biweekly 30 minute program bringing you interviews featuring bay area scientists and technologists as well as a calendar of local events and news.


Speaker 3: Good afternoon. I'm Rick Karnofsky. Brad swift and I are the hosts of today's show. We're speaking with Jessica and Bradley Vojtech. Jessica is a designer and a developer who earned her master's of information management and systems here at cal and has [00:01:00] worked on several UC Berkeley websites. She's also working on the future of science education through projects like ned the neuron. Brad is in NIH, N N I g m s postdoctoral fellow at ucs f. He got his phd from cal. He's a prolific blogger and Zombie expert. The void techs are here to talk about brain systems, connections and associations and network relationships or brain scanner. This website helps people explore [00:01:30] how neuroscience terms relate to one another in the peer reviewed literature. They've documented their project in our recent journal of neuroscience methods paper.


Speaker 4: Brad and Jessica, welcome to spectrum. Thank you for having us. Thanks. Can you tell us a little bit about brain scanner? Actually, I was at a conference here at cal hell by the CSA, so the cognitive science student association and Undergraduate Association here at cal that they had several neuroscientists and cognitive scientists come and give presentations and I was one of those people. I [00:02:00] was on a panel with a Stanford cognitive scientists at the end of the day. It was a Q and a. We got into a question about what can be known in the neurosciences and I had mentioned that the peer reviewed neuroscientific literature probably smarter than we are. There's something like 3 million peer reviewed neuroscientific publications and I was saying that that is just too many. There is no way for anybody to to integrate all of those facts and I said if there some automated or algorithmic way of doing that, we could probably find some neat stuff out and he disagreed with me pretty strongly on the panel [00:02:30] and I sort of stewed on that for awhile.


Speaker 4: That ended up becoming the brain scanner project actually, which is using text mining to look at how different topics in the neurosciences relate to one another. We had conversation about this and I had just started about six months before my a master's program at the School of Information. So all of the stuff that he was saying really jived with what I was learning. So we got together after that. We talked about it off and on sort of over dinner and stuff occasionally, but I think it was [00:03:00] right around won't. Right, right before we found out you were pregnant. Right, right around Christmas when we first actually sat down together to work on it and that was just a random evening. We didn't have, well, we didn't have a baby at the time, so we didn't have much else to do. Brad was working on this thing and he said, you know, I've been working on this all day.


Speaker 4: I'm trying to get this algorithm to work and see if we can get any results out of this. And I kind of challenged him. I said, I can do that faster than you can started taking my course. I had [00:03:30] all of these new skills that I just wanted to kind of show off. And I did. She actually beat me. You guys were both sort of where we were. We're basically coding. Yeah. We're sitting on the couch. Not really cause we weren't actually doing it together. We are using two different competitor competing. Exactly. So who do you see as the audience for brain scanner? Well I know the answer to that someone. Right. So I have colleagues who tell me a lot of Grad students actually mostly a who say that they use it as a stop for [00:04:00] searching. The peer reviewed neuroscientific literature. So pub med, which is the surface run by the National Library of medicine, which is part of the national institutes for health index is a lot of these peer reviewed biomedical journals.


Speaker 4: Their search engine is quite good but it returns just textual information. You know, you just, you see the 20 most recently published papers or you know, however you want to sort it related to the search term or of interest. Yeah. So basically anybody who wants to create an app can get access to this data. You have to follow certain [00:04:30] rules, but otherwise you can get the information out of this database easily. In a, in a sort of standard format, we provide a graphic or a visualization layer on top of the search so you can put in one of these search terms and you can see here are the topics that relate to it very strongly in literature. Statistically speaking, you know, uh, by that I mean here are the words or terms that show up a lot in papers with the term memory for example. We also then list the papers that are related and you can see the full list of terms and [00:05:00] how it relates to different topics and things like that. Um, if I want to look at a brain region and say, okay, what are the other brain regions that are related to this can really quickly visually see that based upon these 3 million publications that we, we searched through


Speaker 2: [inaudible]


Speaker 4: you are listening to spectrum on k a l x Berkeley. We're talking to with user interface developer, Jessica Vojtech. And neuroscientists, Bradley Wojciech about brain scanner


Speaker 2: [inaudible]


Speaker 4: [00:05:30] do you see other potentially valuable ways you can harness PubMed's data or other reference sources? Yeah, absolutely. So one of the aspects actually of the paper that we published was ways to address that, that very question. Uh, initially we tried to publish the paper just as a here's a, here's a resource or one of the editor's version on that rejected the paper, said, you know, what, what can you do with this? And a, of course, you know, this is something we've been thinking about. And [00:06:00] so I tried to build a proof of concept. So one of the, one of the things that we showed statistically speaking that you can do with this, does the data they call hypothesis generation or semiautomated hypothesis generation. And this works off of a very simple idea. Um, it's almost like recommend their algorithms and um, like linkedin or Facebook or something like that.


Speaker 4: You know, it's like if you know this person, you might know this person, kind of a friend of a friend should be a friend idea. You know, Rick and I, I know you and you know Rick, maybe you have a friend named Jim. And so statistically speaking, [00:06:30] Jim and I might get along right because you and I get along and you, and he'd get along, especially if I and Jim get along. And so you can go through algorithmically and say, you know, in the literature if Migraine for example, which is the example you give in the paper, uh, is strongly related to a neurotransmitter Serotonin, which I didn't know before we made the website actually, um, that in the medical literature there's a whole serotonin hypothesis from migraines I guess because Migraines respond to, uh, antidepressants, which are usually serotonergic drugs. So anyway, Serotonin [00:07:00] and Migraines are very strongly related and neuroscientists know a lot about the basic physiology of Serotonin, where in the brain is expressed and things like that.


Speaker 4: And so on the neuroscience side, we know that Serotonin is very strongly expressed in, in a brain region called the striatum, which is sort of deep frontal part of the brain. And, uh, there's thousands of papers that talk about Serotonin and Migraines and Serotonin in this brain region, the striatum respectively, but there's only 19 papers or something like that to talk about that brain region and migraines. [00:07:30] And so statistically speaking, maybe we're missing something here, right? Maybe just nobody's really looked at this connection between migraines in that brain region. Maybe there aren't papers published on it because people have looked and there's nothing there. But, uh, that's why it's somewhat automated. You can go through this list of recommended hypotheses and you as an expert, I can go through that list and say, oh, some of these are nonsense. Or Oh, that's, that could be interesting.


Speaker 4: Maybe. Maybe we should look into that. So it gives you a low hanging fruit basically. Yeah. And so that would be something [00:08:00] eventually I would like to build into the site. Are you continuing to analyze new papers as they enter in pub med? We haven't rerun it for awhile. I think there's something on the order of 10,000 new papers published every month in the neurosciences. But when you're standing in the face of 3 million, it's sort of drop in the bucket. So we, we worry running it every month or two. Um, but the results really don't change very quickly. Right. It's pretty stable. So yes, we, we should actually [00:08:30] run it again. It's been about six months or so. If you guys actually like, well I mean as a or perhaps how, you know, the ideas in the literature might change. For instance, that's actually something that I did do.


Speaker 4: Um, I eliminated the searches to just bring the regions, so how different brain regions relate to each other across time. So I did a search for all papers published up to like 1905, which wasn't very many. Of course not in your, you know, you have an exponential increase in the number of being published. Okay. But then again, I ran it again for all papers published to like up to 1935, [00:09:00] 55, 75, 95 and you know, 2005, right? Uh, or 2011. And you could actually see how our understanding of how different brain regions relate change over time. And that was kind of neat. Um, if I was going to be a little bit statistically, uh, stronger about this, what I should have done in the original paper, and I didn't think about it until after we republished it was I should've run the semiautomated hypothesis generation algorithm, uh, on a limited amount of data. So I test data set up to like say 1990 [00:09:30] and then found plausible hypotheses from that Dataset and then run it again on the entire thing and see, you know, if we had found new things. And you know, if that corroborated what we've learned in the last 22 years.


Speaker 2: [inaudible]


Speaker 5: you're listening to spectrum on k a l x Berkeley. We're talking with Brad and Jessica Vojtech about brain scanner. They're a site to show links [00:10:00] that may exist between brain structures, cognitive functions, neurological disorders, and more as data mined from the academic literature.


Speaker 2: [inaudible]


Speaker 4: I mean this is a side side project for us. Yeah,


Speaker 1: it was two weeks in $11 and 50 cents. And what did that go to? Um, coffee and coffee. Yeah. [00:10:30] Um, no it, it went into the Google app engine server time. So we actually were able to use Google app engine to distribute the processing, which is also what made mind my code a little bit quicker to run through all of this data.


Speaker 4: I was doing single queries at a time and because we have 800 terms in the database and we have to do how every term relates to every other term, it's 800 squared,


Speaker 1: try to buy two essentially. And then there's the roundtrip between between his your machine and the um, [00:11:00] pub met database. So, you know, you're making requests, you're making requests, making requests anyway. It was maybe three days, three days or four days. And I was able to do it in about two hours by um, putting it into the cloud and using app engine. So that $11 and 50 cents went to paying for the service and agree to say a hundred squared divided by two minus 800 a lot. So do you want to talk about how that dictionary of keywords was generated?


Speaker 4: Initially I [00:11:30] had wanted to try and figure out how brain regions relate this. This grew out of my phd work actually at Berkeley. I worked with Professor Bob Knight who used to be the head of the neurosciences institute, Helen Wells neuroscience here. And my phd thesis was looking at how to brain regions, the prefrontal cortex and the Basal Ganglia related to working memory. And as I was standing for my qualifying exams, I was trying to figure out, okay, what are the brain regions that send inputs to [00:12:00] [inaudible], which is one of the parts of the Basal Ganglia and where he dies this ride in project two. And in order to figure that out, I spent, I don't know, two months off and on three months off and on over at the biomedical library here, digging through old, uh, anatomical papers from the 1970s and basically drawing little hand-drawn charts to try and figure out how these things connected.


Speaker 4: And it really surprised me. It was frustrating because you know, here we are in, well, when [00:12:30] I was doing this, it was like 2008 right? And all I wanted to know is how different brain regions connect. And I was like, why can't I just go to a website and say, okay, striatum, what are its inputs and outputs? Like we have that information, right? Why can't I do that? Um, and so anyway, that was one of the motivating factors for me also. And there's actually a paper published in 2002 called neuro names. And then this researcher was trying to create an ontology of, of brain region names Ryan. So the terms that we use now in 2012 aren't necessarily [00:13:00] the same that people were using back in 1900 when they were first describing the basic anatomy. And so you have some Latin names for brain regions.


Speaker 4: You have modern names for brain regions, you have names for different groupings of brain regions. So I referred earlier to the base like Ganglia, uh, and that is composed of, you know, maybe five different brain regions. And if I talk about three of those brain regions, uh, can I give examples? Is the putamen and the Globus Pallidus, uh, Globus Pallidus is actually contained [00:13:30] of two separate ones. And the putamen and Globus Pallidus if you combine them together or known by one name. But if you combine the putamen with the striatum, that's a different name. And so you actually have these weird venn diagram overlapping naming Schema.


Speaker 1: There's a significant vocabulary problem, which is the term that we use in the information sciences. Basically the fact that you have multiple names for the same thing and you have the same name for some different things. So you know this venn diagram idea. Um, so yeah, [00:14:00] if you're going to use a very simple search algorithm, you have, you can't do it, you wouldn't, you're not going to get all of the results. So, um, I think our system tries to solve that vocabulary problem a little bit.


Speaker 4: And then there's actually a researcher, um, Russ Poltrack drag, who used to be a faculty of neuroscience at UCLA and I think he's in University of Texas now. And he actually tried to create an ontology for cognitive term. So in cognitive science and psychology and cognitive neuroscience, you know, we have terms like working memory [00:14:30] and attention and in they're trying to create a whole ontology for how these different things really. So like working memory as part of memory, which you know, in memory also contains a longterm memory. And so we'll use his first attempt as a dictionary as well. And then we went to the NIH website and they've got a listing of all these different kinds of neurological disorders and we use that. So we pulled a bunch of publicly available data basically and use those dictionaries as our starting point.


Speaker 1: And then we [00:15:00] also took suggestions from the people on our website almost immediately we started getting requests for more and different terms. So you had the, when you find two keywords that appear in a paper together, you assume that they're actually related. Can you comment on if people might have demonstrated that they're not actually related, how does that affect your system? Like some, like an instance in which, uh, it says this brain region is not connected to this other [00:15:30] brain region, right? Um, yes, we have assumed that there's a publication bias that if there is not a connection then someone does not publish a paper about that.


Speaker 4: Okay. And negative publications or negative findings go very under reported in the scientific literature.


Speaker 1: Right. So we're hopefully taking advantage of that. Hopefully the law of large numbers means that our data is mostly correct and it does seem to be that way. The example that Brad gave, uh, with the Allen Brain Atlas, [00:16:00] that there is certain corroborating evidence that seems to suggest that this is a, at least plausible connections. There's obviously no one say that better. No, that's perfectly scientifically accurate. I tend to get a little bit specific when I'm talking about this kind of stuff.


Speaker 4: Is there already some sort of bias that might drive certain kinds of papers up? If the paper has a lot of buzzwords, perhaps it suddenly becomes more important. Do you 100% yes, absolutely. There are always [00:16:30] hot topics. Uh, and that shows up for sure


Speaker 1: only because there's more papers published on that subject. We don't currently have a any kind of waiting per paper.


Speaker 4: Yeah. Like when you go into the website and you'd do something like, um, there's a brain region called the Amygdala and you know, it'll be very strongly associated with fear. And so that's actually one of my concerns is problem getting these biases. So, you know, there's a lot of literature on this brain region, the Amygdala and how it relates to fear, but it certainly does a lot more than just processing fear, [00:17:00] right? It's this general emotional affective labeling sort of idea that anyway, that's, that's neuroscience specific stuff, you know, and brain region called the insula and disgust or love or you know, these other kinds of strong emotions. And so yeah, it definitely reflects certain biases as well. And we, we try and quantify that even to an extent a little bit. So again, using the Allen Brain Atlas data, we show from our Dataset, what are the top five brain regions that express or that are related to dopamine, for example.


Speaker 4: And in the real human brain, what are the top five brain [00:17:30] regions that express dopaminergic related genes? And you can actually see that there's a very clear bias. So one of the regions that expresses dobutamine very strongly is very hard to study. Neuroscientifically speaking. It's, it's deepen deep part of the brain. It's hard to get any, it's very small, so you can't get it from like brain scanning expresses a lot of dopamine, but people don't study it and we can actually quantify then some of these under-studied relationships, right? We're like, here's a brain region that we know expresses a lot of dopamine, but there's a a hundred papers only and another [00:18:00] brain region that's very sexy and too about domain has 10,000 papers. Right? So our system shows you an example well of the current state of scientific literature. So it's not necessarily 100% correct, but it reflects what scientists think as a whole at this point. Yeah, I agree. And we try and be very careful about that in the paper and in talking about it like we are right now


Speaker 2: [inaudible]


Speaker 5: [00:18:30] you are listening to spectrum on k a l x Berkeley. We're talking with user interface developer, Jessica Vojtech and neuroscientist Bradley Vojtech about brain scanner.


Speaker 2: [inaudible]


Speaker 4: I was really surprised you. I taught neuro anatomy for three semesters here at Berkeley and you know, so I know the anatomy pretty well. And on your first ran it, I had one of those like yes, kind [00:19:00] of moments like I can't believe this work because it really does find all of these clusters really nicely. And that was a very pleasant surprise because technically speaking it couldn't have been any other way. Like it just has to, you know, I mean these topics co-occur a lot, so it should be that way, but it's always nice to see something like that work. Brian, I wanted to ask about the journals that you sent the paper off too. How did you pick them? Art Of picking a journal where to send a paper. It's actually really hard. So certain journals get [00:19:30] more readership than others. And then there's the open access factor.


Speaker 4: So I'm, I'm a big open science, open data advocate and so I try and shoot for that. I had forgotten, there's actually sort of a, a very wide protest of Elsevier, which is one of the publishing companies right now. And the journal that published my papers and Elsevier Journal, but, uh, I had signed the petition and I was part of that Nash shortly thereafter. That would have impacted my decision had I been thinking about it. Yeah. And yeah, so it's mainly a balance between readership and expectation and you sort [00:20:00] of get a feel after publishing a few papers of what editors are looking for. And so yeah, I am the one that has experience with navigating the academic publishing environment. Yeah. So yeah, we sent it out to a lot of journals and, uh, mostly it didn't pass editorial review, which means that there's an editor that decides whether or not conceptually it will be interesting for their journal to publish it once got center review at a journal and they're like, well, it was sort of torn.


Speaker 4: There were four reviewers, four pure reviewers, [00:20:30] and two of them were fairly enthusiastic and the other two are like, this is cool, but so what? Right. Um, and the general consensus was it didn't fit with the theme of the Journal. The Journal of neuroscience methods point really well and your reviewers are very, and um, actually there's a figure at the end of the paper where we did some integration with the Allen Brain. Alice Paul Allen, one of the co founders of Microsoft who is a cuisine heir, has put half a billion dollars into this institute. [00:21:00] Initially the goal was to map, uh, the expression of all of these different genes in the human brain, in the mouse brain, and they made all that data publicly available. And so we use that as a test data set. So we said, okay, where are these different, uh, neurotransmitter related genes actually expressed in the brain and what does our system think about wearing the brain? These neurotransmitters are, there's a week but significant correlation between the two, which suggests that our system reflects actual reality to a certain extent at least. [00:21:30] And that was a suggestion I got from one of the peer reviewers and that was really good. It was a lot of extra work, but it ended up being a really good addition to the paper.


Speaker 1: But both of you guys are involved in science education and science outreach. So I was hoping you can comment on that. I'm actually starting a project with a friend of mine building a neuroscience kids books. So we're going to teach neuroscience to elementary school kids with an electronic ebook featuring the neuron. Yes, featured his name is ned the neuron. He's a pure middle cell and he works in the motor cortex of the brain. [00:22:00] And is the neuroscience focus partly driven by bad or do you have any sort of personal interest in as well? I do have a personal interest in and I, I, you know, obviously it's convenient that my husband is a neuroscientist, but actually the character and the original story idea is my partners who's also a neuroscientist and phd in neuroscience here at cal here at cal.


Speaker 4: Yeah. I get this question a fair amount. Like why do I do blogging and outreach and things like that. So there's actually a few answers to that. One I find blogging, uh, helps me [00:22:30] do better science. If I have to figure out a very simple way of explaining something, then I feel like I understand it better. It's sort of like one of the best ways to learn something is by trying to teach it. Right. I had a very strange path to academia. I actually got kicked out as an undergraduate from the university. I had to sort of beg my way back in because my grades were pretty low. You know, a couple of people help me out along the way and that were pretty important to me. And I think a lot of Grad students have this experience where they, they feel like they don't belong there in in sense that like, oh [00:23:00] my God, I'm not smart enough to do this.


Speaker 4: You know? And when I look at the resumes or cvs of, you know, tenured faculty here at Berkeley, right? It's just paper after paper and award and amazing achievement and you're just like struggling to even understand how to write a paper and it seems just like this daunting, intractable problem. And so because of that, I actually have a section in my CV where I actually list every time a paper has been rejected. I've actually had graduate students tell me that. That's been kind of Nice to see that you know, you see somebody who's doing pretty well and you see that, you know, in order to get there [00:23:30] you sort of have to slog through a lot of crap.


Speaker 1: Did you plan to work together some more? I think so. You know, we're obviously working together to raise a son right now. We actually were talking on the way over here about trying to implement some of the ideas we've


Speaker 4: been talking about that people have suggested. I think we could definitely do that. Yeah, there's definitely a lot of overlap. I'm very interested in dynamic data visualization and that's something that Jesse's is obviously getting quite quite good at and so I'd [00:24:00] like to start doing that for a lot of my research papers as well. Brad and Jess, thanks for joining us. Oh, thank you very much for having us. Thank you so much for having us.


Speaker 2: [inaudible]


Speaker 6: and now for some science news headlines. Here's Renee Rao and Brad sweet


Speaker 2: [inaudible]


Speaker 7: [00:24:30] the Berkeley new center reports researchers at the University of California Berkeley are gathering evidence this fall that the Feisty Fox squirrels scampering around campus or not just mindlessly foraging for food but engaging in a long term savings strategy to track the nut stashing activity. The student researchers are using GPS technology to record all of the food burials and in the process are creating [00:25:00] an elaborate map showing every campus tree building and garbage can. Miquel Delgado a doctorial student in psychology heads the squirrel research team in the laboratory of UC Berkeley, psychologist Luchea Jacobs. The research team is replicating the caching experiment on humans by timing students as they burry Easter eggs on campus and try to find them. We're using humans as a model for squirrel behavior to ask questions that we can't ask. Squirrels still got us said the group has a cow squirrels website to promote their work.


Speaker 6: [00:25:30] UC Berkeley professor of cell and molecular biology and chemistry. Carolyn Bertozzi has won the 2012 Heinrich Violin prize. Professor Bartow Z has founded the field of bio orthogonal chemistry. In her groundbreaking approach, she creatively exploits the benefits of synthetic chemistry to study the vital processes within living beings. Professor Dr Volk Gang Baumeister, chair of the board of Trustees of the Heinrich Violin Prize says of Professor Berto z. [00:26:00] Her breakthrough method to identify sugar patterns on the cell surface is a milestone for our understanding of the functions of sugars in health and disease and paves the way for novel diagnostic and therapeutic approaches.


Speaker 3: Irregular feature of spectrum is a calendar of some of the science and technology related events happening in the bay area over the next two weeks. Brad swift and Renee Rao join me for this. The second annual Bay Area Science Festival is wrapping up this weekend. [00:26:30] Highlights include art in science and gallery gala showing the intersection of image and research tonight at the Berkeley Arts Festival, Gallery Science superheros tonight at the Tech Museum in San Jose and discovery days at at and t park tomorrow November 3rd from 11:00 AM to 4:00 PM last year more than 21,000 people showed up to this free event this year. There are more than 150 exhibits. Visit Bay area science.org for more information about any of these [00:27:00] great activities and to see their regular calendar of science goings on.


Speaker 6: Big Ideas. Berkeley is an annual innovation contest that provides funding, support, and encouragement to interdisciplinary teams of UC undergraduate and graduate students who have big ideas. The pre-proposal entry deadline is 5:00 PM November six 2012 all pre-proposals must be submitted via the online application on the big ideas website. Remember there are big idea advisers to help students craft [00:27:30] their pre-proposals. You can drop in at room 100 Blum hall during scheduled hours or email advisers to schedule an appointment at another time. Check the big ideas website for advisor times or to make an appointment. There will also be an editing blitz November 5th from five to 8:00 PM in room, 100 of bloom hall advisors and past winters will be available to provide applicants with valuable last-minute insights and advice on your pre-proposal. This is a great opportunity to hone your proposal and get support from those [00:28:00] who know what it takes to build a successful big idea. The big ideas website is big ideas.berkeley.edu


Speaker 7: on November 8th the center for ethnographic research will hold a colloquium to understand cancer treatment trajectories using an array of ethnographic data. The Speaker Daniel Dohan and associate professor in the Phillip r Lee Institute for Health Policy Studies. We'll discuss this research about inequality and culture with a focus on cancer. He will focus on his most recent study which examines how patients [00:28:30] with advanced diseases find out about and decide whether to participate in clinical trials of new cancer drugs. The event, which is free and open to the public, will be held from four to 5:30 PM at 25 38 Channing way


Speaker 2: [inaudible]


Speaker 5: the music you [00:29:00] heard during say show was spend less on and David from his album book and Acoustic, it is released under a creative Commons license version 3.0 spectrum was recorded and edited by me, Rick Karnofsky, and by Brad Swift. Thank you for listening to spectrum. If you have comments about the show, please send them to us via email. Our email address is spectrum dot k a l x@yahoo.com [00:29:30] join us in two weeks at this same time. [inaudible].



Hosted on Acast. See acast.com/privacy for more information.

  continue reading

78 episodes

All episodes

×
 
Loading …

Welcome to Player FM!

Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.

 

Quick Reference Guide