Event ID: 3103165
Event Started: 11/2/2016 8:30:21 AM ET
Please stand by for real time captions.
[ Captioner on stand by waiting for event to begin. If there has been a change, please contact Caption Colorado at cc@captionedtext or 800-590-4197. Thank you]
We are going to get started. But I love the background music.
Good morning everyone. And welcome to the AHRQ meeting. This is sadly the last meeting for David Ballard. We are going to miss all of you tremendously. And we will probably reach out to find you again. I want to welcome the members of the National Advisory Council . And those who are on the webcast. Just a couple of housekeeping notes, those of you who are retiring. With no benefits.
[ Laughter ]
We would like to have your photo taken with Andy. And for those who would like to make a public comment. We will have two opportunities. And if you would like to make a comment please sign up at the registers -- registering table. We will also offer meals until 3:00. Now please go around the room and introduce the folks that are here. And please make sure you'd use your microphone so those who were on the webcast can hear you.
My name is Jamie Herman.
My name is Sharon Arnold.
My name is Jeff Weisberg.
My name is Sherry Davidson the --
My name is Jose [Indiscernible] .
My name is Mary Ferguson.
My name is Paul Sherman.
My name is Alice Fast.
My name is [Indiscernible] and IM here representing CDC.
My name is Naomi [Indiscernible] IM from the research and development group.
Good morning. My name is Francis [Indiscernible] .
My name is Dave Knutson.
My name is Andy Bindman.
Who do we have on the telephone ? I know that Don Goldman called in.
My name is Don Goldman. And I am from [Indiscernible] Children's Hospital .
My name is Mary [Indiscernible] .
Is there anyone else on the telephone ? I know that people will be calling in. We have Christina Calla tomorrow -- Callmorrow .
The first order of business is to take a look at the draft minutes. The copy of those minutes should be in your folder. It is a wonderful summary of the meeting. I want to thank Julie for that.
This is David Ballard. I am on the phone.
Wonderful.
Do we have a motion to approve the minutes?
I will second it.
All in favor?
Now I will turn this over to Andy Bindman .
Thank you for those who are able to be here. It is a pleasure to welcome you back to AHRQ . I want to start off with a little humor but I understand my slides were leaked to their press. [ Laughter ]
I want to acknowledge all of the good efforts that were involved here at 5600. But anyway, thank you for being here this morning. So what I am going to do is give you a first overview on the agenda for today. And I will also give you the directors update. And then we have three different presentations. And this is quickly going to be that defined theme that we will talk about today. And after you hear the three presentations, you will be able to ask specific questions. Then we will take a lunch break. Then we have a panel. And this is to create an opportunity to reflect on what you heard. And to get you thinking about issues, and what is going on at AHRQ . And so that is really our focus. Then we will have a discussion and then we will have a final wrapup. We will have a focus meeting on a particular theme. And we want to make sure that you are prepared for that. And this is your chance to be engaged. So let's first get through the update information. So you have a sense of the progress that we have made. On things that we discussed and other notable things for AHRQ . I want to commend Beth who had a promotion. She is now vice president at [Indiscernible] at Kaiser Permanente. That is so impressive. But anyway, congratulations. And the best of luck. And we look forward to hearing about that. We are sorry about the departure of David Ballard, Sherry [Indiscernible] and Henrik Sherman, Jeff Weisberg. Thank you, and you will always be a part of AHRQ family. We hope to continue to have ongoing contact with each one of you. Let me get into some of the things that we talked about from the last NAC meeting. We talked about the evidence-based practice. Arlene [Indiscernible] gave you an overview of those at cavities and what is going on in NAC and activities . And how to shape this program moving forward. And some of the things that we have taken in terms of action since that meeting. We have identified, the enhancements of the application of the review. And the goal is not for AHRQ to be in a place that is harvesting best evidence. But we want to share the evidence. So we are working on identifying influential stakeholders, and to share the evidence base practice. We are in the process of engaging with stakeholders and getting their feedback about the usefulness of other reports and how our partners use these reports and we are incorporating this into our assessment. And we want recommendation about the future [Indiscernible] . And people have made recommendations to us. We want to use this to help us amplify and so forth. So we have a strong sense on the science and the actual dissemination once it is put together. And we have really identified one group, that we have engaged in this process. This is our health systems and this is what we are going to talk about today as we engage more strongly with the system and look at what they recommend to us. And we see them as a potential important target. In terms of making use of this work. There is specific enhancements that we have been working on. To make it more feasible. For instance the team MessageBox. We wanted to create a simple way to communicate. And we have also establish a quality improvement team. Looking into our whole process looking at how long it takes us. And the execution of the review and so forth. We want to put in more efficiency in that process and streamlining is part of that process. And so we are working on the process as well as connecting the results and how they are impacted. And the other topic that we talked about at that last NAC meeting, was on quality measurement. One of the things that you recall, we talked about the range of activity that AHRQ is involved with . This is a quality measure, testing them and ultimately moving into stewardship for the ongoing application. We did have a discussion here about as a research organization that we should prioritize [Indiscernible] . And how we can think about the different stages of development. And since our last meeting, we are really doing our own assessment on quality measures. We in a sense take ownership here at AHRQ . And think about the ones that are at a stage in which they are being applied in used -- and used. And is there a relationship in [Indiscernible] and handing it off ?
We are investing on the research, and development. And we are also thinking strategically about new storage data that is going to be important. Things like electronic health records. And we want to identify partners that will work with us in that area and also develop measurements. And this is evolving as a very robust program. We want to have the capacity to build on that. And reaching into new sources of data. And the role that we can play in quality measures in that realm. We also did talk about, the work underway and the responsibility for AHRQ . If you recall we have a mandatory allotment of funds, and a trust fund. And this is for the purpose of dissemination and implementation, not only from NIH and VA but [Indiscernible] . And we have evidence around PCOR and we are trying to align our activities. And we thought it would be value -- valuable to create a subcommittee. And this was a development of this committee, to work on this particular topic area. It is my understanding that the subcommittee will provide advice. And they will coordinate the work that AHRQ did with other federal agencies. I want to thank Mary, who agreed to chair. Even though we do not provide food here at AHRQ . Trends -- [ Laughter ]
We will have a report at the NAC meeting. And the first report out will be the next meeting. We want to thank NAC and its members for serving. I also want to tell you about some thing that has gone on since the last time we met. In September we had a very successful meeting. It was a research Summit on diagnostic errors we had to wonder people and 100 on the -- we had 200 people here and we had 100 people on the webcast. We had people from Germany and all over the world. We did see several presentations. And the webcast was done live, and a recording of that will be available soon on our website along with the slides. I want to say that it was a wonderful event. It was great to feel -- having this wonderful committee join us in our home. We did have a slide about how many babies were born. And it was great. We had people coming in from the research community getting engaged around this issue and this allowed people to get a sense of where we live and do our work. We were enjoying a lot of the exchange. And we were interacting with many individuals. And this was a wonderful way to bring the research community together. This is one that we are very excited about I wanted to remind you about the national Academy [Indiscernible] . About a year ago we had [Indiscernible] , and one of the speakers asked at the meeting, about the history of [Indiscernible] and handing it off. So that AHRQ can develop the research agenda . So it was a very successful and exciting meeting. I think sometimes. As we talked about this at the meeting, many times it is estimated that almost everyone, at some point in their life will be a victim of diagnostic error. It is estimated to be 10,000 times a year. And when you have a diagnostic error, or a challenge in diagnosis. The benefit to harm, can really change.
[ Captioners Transitioning ]
This happened tens of thousands of times and when you have diagnostic errors and challenges in diagnosis, the equation of benefit to harm that can come from medical care that follows from that really changes. You are much higher probability of introducing home relative to benefit. And if so, this is a very important area, and we went into and highlighted the role of tools that can support cognitive improvements, issues that need to be addressed in terms of systems, and the role that IT can play in supporting improvements and improving diagnosis. We also shared that AHRQ has put forward two funding announcements already in this area as part of our work of expanding our work in patient safety, into this area. One of the calls for funding is an R01 type in which it is to prove ambulatory care, what are the contributing factors, and we are also using an R18 mechanism which is really much more around, okay, if we have ideas about what the problem is, what are the tools they need to be developed to actually address the problem? The R01 is much more on the let's get evidence about what the issue is and the R18, as we are starting to know about that, how do we apply ways of addressing it? This is typical of how AHRQ historically does its work of trying to identify research, findings and evidence but also to make sure that we have a dissemination and implementation plan in mind as well.
I now want to shift and talk about we have closed fiscal year 2016. As he recalled the federal government uses a different calendar, then we start the federal year on October 1 each year. And so, I wanted to give you a sense of some of the spending trends in terms of investigator initiated and targeted research, support from AHRQ, and, this includes both the grants that we were able to support from the appropriated funds from Congress as well as funds that we obtained from the PCOR trust fund. And, in 2016, we had $189.6 million in grant funding from those two sources. In 2017, we don't know, get, for sure, as you recall, we are on a continuing resolution from Congress that will get us through, I believe, December 9. And following the election, it is anticipated that there will be some further consideration of the budget for 2017, but at this point, we are projecting having $192.3 million in grant funding available to us, and you can see how that compares to 2015 and 2014. You recall that between 2015 and 2016, the agency as a whole had an 8% cut against funding from the appropriations process. In terms of where we are in the 2017 budget requests, the 2017 discretionary request was for $363.7 million, this is broken out, both 280 million that would come from a budget appropriation as well as $83 million from something called the PHS evaluation funds, which is a mechanism within the Department of Health and human services to do some reallocation of funds among the different agencies.
If that were to be enacted, that would be a $29.7 million increase from the fiscal year 2016 amounts that we received, but, the total amount would actually be $469.7 million because of also anticipated increase is in the PCOR trust fund, mandatory amounts that we would receive there. And that is noted there at the bottom. That we are anticipating the possibility of 100 and million dollars in mandatory funds from the PCOR trust fund, which would be a pump of about $11.5 million from the prior year.
In terms of what Congress has done to date with regard to setting a budget for AHRQ, in July 2014, the health of Appropriations Committee made a recommendation of $280.2 million for fiscal year 2017, which would be an approximately $54 million less than what we received in 2016. This is an amount obviously considerably lower than we had, that the presidents -- president's budget had called for. I will say that to some extent, we saw this as a little bit of maybe the silver lining associated with this. It was the first time I believe in 4 years we did not get a zero from this committee.
[ Laughter ]
I don't know what the percentage increase from zero that is but it's very hard to calculate. Very hard to calculate. On June 7, the Senate subcommittee on labor, health and human services represent -- recommended $324 million which would be a $10 million decrease from 2016. And again, 2016 was an 8% cut from 2015. And as I already reference, there is a continuing resolution through December 9, which basically prorate forward our 2016 budget. There is a small .496, for those who are calculating at home, reduction across the board for the budget, not specific to AHRQ. So, because we are uncertain of course what will happen beyond December 9, we have really been trying to be cautious in how we use our funds and we are really only providing necessary and limited funding in order to provide continuation of projects and activities. And, kind of going a little bit more slowly in basically obligating future spending until we know more for certain about what our ultimate 2017 budget will look like.
In terms of what I am trying to do in response to some of the issues that I have learned about with regard to why it is that AHRQ continues to have challenges receiving the budget, that the president has requested on behalf of AHRQ, I think part of it is making sure members of Congress have a good understanding of what AHRQ's work is and what difference it makes. And if so, I continue to spend time meeting with members of Congress and their staff to educate them about AHRQ's activities. And so, I have found that to be something that I have found that there have been very good exchanges there. There has been a lot of interest and openness to hearing about the activities here, and, there have been good questions raised in that context. And so I think that has been something that has helped to communicate to members of Congress about the work we are doing here and I think that is going to be an ongoing need, to make sure that not only AHRQ is tough but that those who are engaged with AHRQ have the ability to communicate that story. So that Congress understands what it is receiving for the investment on behalf of the public. I am also working with partners that have been identified as working in areas that are highly related to what AHRQ does. For example, meeting with the NIH, CMS or their innovation center, CMMI, meeting with PCORI, and really talking with anyone, their leadership, around the roles that they play that have relationship to the work that we do here at AHRQ and making sure there is Arity about what our role is, relative to their role, and making sure that we are not being redundant, but are being aligned. Again, I found that to be very productive and constructive communications with all of those groups. And then finally, also really working in tight coordination with our staff here at AHRQ to do internal review of what our work is, and making sure that we are developing the knowledge we need to have about all of our programs and a shared understanding of what those are and what the impact of them is. So that we have clarity about what we believe the mission is that we are trying to accomplish and to make sure that our programs and projects are well aligned toward that. And, I have found that to be a very cooperative spirit here at AHRQ around all of that.
One of the things that I have emphasized is that I think AHRQ in self not only needs to be a great supporter of new knowledge with partners that we fund, but that we also need to make sure that we are continuing to function as a learning organization so that we want to develop better ways to measure the impact of our work, and be able to use that as a way of continuous feedback to ourselves about how we can enhance the impact of our programs. I really want to make sure that we are in a using data and feedback about impact as a way to continually improve ourselves, to improve the effectiveness of our programs, to improve the spread of our programs and ultimately make sure that AHRQ is making an important contribution to improving our healthcare. And, this is such an important topic that I have earmarked it as a possible topic for our upcoming NAC meeting. How could we gather input and think about ways to measure impact in a consistent way over time, and what are the new models that we might use to disseminate our work so that we can expand our impact? So, this is something that I'm thinking about for an upcoming meeting. Sandy, you have your hand out but you should also introduce yourself because I think that you did not have a chance to do that before.
Turn it on.
Five hours to get from Philadelphia, I could have walked. [ Laughter ] the efficiency of Amtrak to the Metro system. I'm Sandy Schwartz, professor of medicine, health management and I am working in school, clinical health service, interested in decision-making and assessment of medical interventions. But, Andy, what I was wondering is, in your conversations with Congress, or with other agencies, but my guess is the main thing in the Congress and in house in particular, is there anything communicated about AHRQ that, you know, collectively, we can work on to educate or respond to?
So, thank you for that question. You know, my sense is that in many cases, the conversations start with a need to want to know more about what AHRQ does. I think that there is among many people a challenge to get beyond the first sentence or two of what they think AHRQ does. You know, many members of Congress and their staff have a sense of some of the work that AHRQ does for safety but do not have a lot of specificity around that, nor do they have a sense of some of the other work that AHRQ does with regard to working with healthcare systems and providers to improve healthcare quality and safety and so forth, related to our mission. I guess one of the things that I have taken away is that there is a need for not only individuals who work here at AHRQ, such as myself, but also for our broader community, to be able to educate Congress about the importance of AHRQ's work and the impact it has had. So, I do think that we have had maybe the size of the footprint of doing that is maybe something that could be enhanced to be sure that people have a broader knowledge. There has been the issue that I think has been surfaced here, that there were also questions raised about, even when the topic is understood with AHRQ works, and is it redundant with some of the other agencies or with PCORI and so forth? That gets more nuanced to be able to help explain how AHRQ's work is aligned and not redundant. And again, that is part of both my attempting to explain that, but also the work that I am doing with other agency heads and with PCORI and so forth, to make sure that what I believe to be the way we are working with alignment is how they perceive it as well. And so that assuming that when they have opportunities to educate members of Congress, that consistent stories are shared. And, that reflects a global understanding of how the work is done together. So, I think it is those issues, both an in-depth knowledge of our work, as well as maybe in some cases, a rapid perception that some may have of, well, that sounds like an activity that goes on in this other part of the government. So, what is the unique role. Although I have had interesting experiences where when I have been told that in some cases about possible redundancy and have pursued on behalf of members of Congress or there staff that I have met with, and try to find out, is AHRQ doing something redundant in another part that I often find it is actually AHRQ's work that has been taken up in another part of the government and then have been able to share back with those members of Congress who were very gratified to hear that. In fact, given our role is about dissemination and implementation, that this in fact is a positive sign. And so, I think that is also something that is very important for all of us to think about. Which is, when does what appeared to be overlap actually reflect a positive thing as opposed to an inefficiency which I think is sometimes the perception that some might have.
Lucy, you have a question on the phone?
Yes, thank you. First, I would just like to say how much I appreciate the energy and proactive attention that I think for your leadership and work that the team has done has really enhanced AHRQ. I have had several conversations from members of Congress and I attended the diagnostic area, some of which I thought was really fantastic. But, my question is, how does our EPC link work or complement PCORI's newly launched evidence? I'm just afraid that it is another thing that will muddy the water from this duplicative work.
Great, Lucy, thank you for the question. So, just to make sure that everybody understands the context of the question. AHRQ has been very well known for its work with evidence-based practice centers, and, we had become aware around the time of the last NAC meeting that PCORI was indicating that it had an interest itself in the role of evidence synthesis, as a way to perhaps find ways to more rapidly identify potential either areas in need of investigation, or that could be potentially from synthesize literature, allow for some of the head to head comparisons of different treatment options for clinical problems. And, as you are raising your question, Lucy, we were made aware of this by PCORI and we are concerned about, with this contribute toward, if we were doing this work and PCORI were doing it separately, the perception that we are in a space in which there is some redundancy that could be going on? So, I engaged in conversations, with leadership at PCORI, and specifically asked what their interest was in this area, asked if AHRQ could be part of the solution for them, rather than a competitor in some ways. We are pursuing, it's not finalized, yet, but are coming along in conversations with PCORI, for them to set up a memorandum of understanding for AHRQ to do some of these reviews on their behalf and areas that they have identified. And so, that would be one way in which they could be working in alignment with PCORI to accomplish an interest that they have using expertise that they had. Having said that, I also am aware that PCORI is thinking about other kinds of ways of doing evidence synthesis, and I am continuing to engage with them to make sure that I'm aware of that, and that we can do that in cooperation with them, or if they are going to work in those areas, that would be important for us to be aware of and reassess here at AHRQ about whether that is an activity that we should continue, or how we can navigate that in a way. Because it doesn't make sense I think to have redundant activities going on. And, as part of our own work here related to evidence synthesis, I am also really thinking about certain kinds of topic areas that I think are less likely, and I don't mean separating stomachache from headache, because I don't think that is enough of a distinction. But, it may be that aligned with some of the interest as an example that we are going to talk about health symptoms -- health systems, that there may be good alignment of our evidence review work on thinking about the evidence around certain kinds of delivery system models or understanding, that certain kinds of unique questions that health systems may have that are different from the systems that PCORI may have around evidence systems. That is a very important question, Lucy, and one that we are cognizant here that we do not want to have in reality, or even in optics, the perception of redundancy. And, I am pleased that we have been able to engage with PCORI in discussions about this, and we will continue to do so to try to navigate the issue you are bringing up.
Rate, thank you.
So, I just want to add that I think this is a topic where we would very much welcome input from the subcommittee of the NAC that has been established. So, we look forward to hearing comments.
I wanted to add one more thing to your list of discussions with PCORI. They recently had a dissemination day where they went through and split up groups on patients, health plan or health partner, and employers, to talk about how to disseminate their work. And, we spent a full day trying to figure out new models and new ideas and I would love for you to just latch onto that instead of, you know, well, in addition to looking for other ways. Because I think it was a valuable day. And, for me, it made me realize because I was in a room of other employer focused groups, that maybe I need my own NAC with that group to say, what else can we be doing? Because there's a lot of synergy in that group as well. I thought it was a valuable day and there is some information there that you should probably take advantage of.
Thanks, Sherry. We did have representation from AHRQ at the meeting and are aware of that. I will say this is another important topic area that I think is important, as you recall from the ACA law, the AHRQ, and the reason we get the PCOR trust funds that we do is to be able to do dissemination and implementation of PCOR findings. We are really trying to engage, again, with PCORI, about the role that we will play in that space and how to align that with them and maybe just to echo Sharon from a moment ago, I think this would be another important area to help give us advice to the NAC about this issue, and, we have, as part of the process, hope to have involvement of PCORI representation on that as well so that we can work toward that clinical alignment. Standing and then they'll me.
Well, along those lines and maybe a little bit beyond that, when it comes to disseminating, it would really be useful I think for AHRQ if, you know, we could have some input into what questions PCORI is going to be asking or how they are going to be asking those questions. Because, are they going to be stuck beyond possibly disseminating something which way -- may not reflect the full range of interest of, say, AHRQ stakeholders, which may be different than the way PCORI is looking at it. And along those lines, I think one thing that may be a real opportunity for synergy might be related to preventative services, under the affordable care act, the task force recommendations if they are A or B automatically. We know that there are always questions on the task force of evidence. Sometimes those are nice to know but sometimes they are fundamental and might make the difference between A and B or another category. That might be an opportunity to identify a common research agenda that would feed into AHRQ or where it might be able to play a larger role.
Thank you for that. I can't remember if it had been raised in this set up before, but, the task force already is quite conscious of those issues of opportunities where they identify sort of missing evidence, if you will, that would affect the grading. There is both a very strategic process with the NIH, as well as feedback to PCORI around that to help inform the opportunities for future research could actually be specifically helpful around those kinds of recommendations. So, I think we are in full agreement of that being a nice way of aligning with the work.
I agree. Almost nobody ever does. Just a thought.
So far, you are doing it. I can't say it will be a permanent thing.
Yes, Hello. This is wonderful and often see, the VA is very interested in hearing more about your collaboration with PCORI. We at the VA, through health services research, we have a small contract. By no means does that compare with what you are envisioning but one of the things, as many of you know, the VA is going through its own healthcare transformation, becoming a pair as well as a provider. And we are in need of rapid synthesis. And so, the contractor that we have, and I am still new there, it takes usually about six months to longer. This is more in terms of how quickly and how rapidly we can get the synthesis, get this information so that we can start implementing it. So that is something that we are in need of and I did not know what your thoughts were, as you talk more with PCORI on the specifics of the collaboration.
As a reference related to some of our work and synthesis evidence, we have a very specific QI process underway to look at, are there ways that we can speed up the process? I mean, just to make it a little more concrete, if there are ways of updating. So, if you have certain questions, are there ways to build upon the platform of previous reviews that allow you to take advantage of that and then just do the marginal evidence since then? Does that speed up the process in some ways? So, I don't want to get too deep into the details without calling upon some of the experts at AHRQ around that to follow up more with you but we are very cognizant of timeliness and an important part of this to make it relevant for the end-users. Because if the end-users have had to make a decision and meanwhile you are going, we are just getting that work. You have missed the train. We are cognizant of that and thinking about the processes that we could use to make sure it is in alignment with what the end-user's needs are. Mary?
Yeah, just going back to the idea of delineating AHRQ's role from the other federal agencies, one of the things that I was thinking about is maybe developing some very short and sweet bulleted items to delineate your role, for example, from CMS. Just to give you an example on quality measurement. Now, for those of us who are in the measure field, we understand what AHRQ and CMS is doing but not a whole lot of people outside of this small circle understand why AHRQ is developing measures as well as CMS. So, just having some common understanding between agencies about what you are doing versus what CMS is doing.
Thank you for that. And I think we spoke a little bit about this at the last NAC meeting, where it is our perception that, you know, CMS has a broad need for measures. And, that is the really good news, that measurement has moved into the sphere of payment. And, AHRQ is one source that can support the needs that CMS has with regard to moving from volume to value in terms of payment. And, CMS has made great use of several measures that AHRQ has developed and as we talked about at the last meeting, the question was, in some of those cases, does not allow for some opportunity for AHRQ to say, we have produced this, you're using it, maybe now CMS will have ongoing stewardship of the ones that are fully ripened and that we should go back and invest our activity in developing new ones come with you sources like EHR data and so forth? So I think your point is right, which is that we need to make sure of our role, CMS making use of what we are producing, sort of like if you were to go to the supermarket, there would not be anything on the shelf if someone did not produce it first but, CMS is absolutely the front end of that as a payer.
You know, more of addressing Congress, for example. When you are going for funding, CMS is already developing measures. The high level of delineating differences in the complementary role.
I'm going to suggest that this question be the last until Andy finishes his presentation, just because otherwise we will be here all night which would be fun.
We have not even left the warning yet. Since Sandy brought up the disseminating task forces and the A and B stuff, maybe this is a good time to ask the question. The task force is very benevolent about the ratings used for criteria of insurance coverage and yet insurance coverage is thought to be one of the principal elements in allowing dissemination. But, we know that the ambiguity in how some of those A and B recommendations is actually applied in medical policies have, for example, in colonoscopy come it took four or 5 years to sort of figure out the details of whether a polypectomy, pathology and those sorts of things are actually covered for no co-pay or not. What is your view on the meaning of the A and B recommendation and further clarification as a mode to dissemination?
[ Laughter ]
That's why it is the last question.
Well, thank you for that.
I should have cut it off before him.
Yeah. I definitely heard the part about getting a polypectomy. But, yeah. So --
[ Laughter ]
So you know, the task force, which we do support here through AHRQ, have an incredibly important role of making recommendations, synthesizing the evidence and forming recommendations regarding prevention in the primary care setting. You know, you made an introductory comment about ambivalence they have and so forth. I'm not sure I would completely endorse that notion. I think there is certainly debate in the broader community about what is currently the law with the ACA, it made it such that as you referenced, R18 recommendations are those that essentially need to be covered as benefits by certain plans and have to be considered by other plans and so forth. And the task force undertakes this work, focused on the research evidence and synthesis on that basis. And, we support them in that way. We are not a body that is designed to evaluate the policy itself and AHRQ does not have a position on that, other than we execute what the law tells us to do with regard to that. So, I think what we try to do is undertake with the best available evidence and as Sandy's question raised, when we identify problems, or shortcomings in the evidence, we make sure that we can work with partners who can help us generate evidence so that the best possible conclusions can be arrived to on a scientific basis.
[ Captioners transitioning ] Because 2016 came to a close, I wanted to give you an update on what funding levels look like in our different funding mechanisms, and I think you can see that AHRQ has been actually -- this is something I've been able to share with study sections, a couple of them were in town last week that work in the study section. The work and study section is no longer a working futility but tremendous opportunity to recognize and support in a grants. Can see that is what the pay lines are as a percentage across the different type of grant mechanisms that we have overall. It was 28% of applications were funded in 2016. I wanted to point out that we have a couple of new announcements that are out to make you aware of. We have a large health research [Indiscernible] Dissemination project for prevention of Healthcare-Associated Infections. That is a R18, Dissemination type mechanism. We have a R01 for Healthcare-Associated Infections. For the R01 and R18 and related area. We also have published a Notice of Intent to publish Funding Opportunity Announcement scop about implementation science and Dissemination implementation study. We are continuing to think about our role in responsibility with regard to the [Indiscernible] trust funds and, more broadly, as will a AHRQ, in terms of our world, how we can enhance the science around Dissemination and implementation. I also want to let you know that since the last meeting, we awarded $13 million to test children's quality measures. This is a collaboration we are doing with CMS. We identified six grantees. If you recall, from our earlier meeting, AHRQ was involved in funding grantees to develop pediatric quality measures. This around the funding is about getting the quality measures used. These are the measures that were developed as part of the develop quality program cost centers of excellence and this is now an exciting next phase of really moving measures to the implementation stage. Again, it's very consistent with how AHRQ does it's work of developing and then implementing. Moving on to the tools and training developments, want to bring to your attention that AHRQ has had a long-standing arrangement and funding in the area of healthcare simulation. Virtually all medical training programs have medical schools -- medical schools of simulators used to help train health professionals about different kinds of healthcare situations, and through work that we have done with the society for simulation in healthcare. We have actually now published a comprehensive dictionary that creates a uniform set of standards of definitions in this area, which we think will provide for a uniform terminology and really clarify communication and clarity for use in teaching, education, assessment and research. I think this is a very nice, again, example of how AHRQ can come in and provide important [Indiscernible] that can advance the field of improving safety and quality in health care delivery. We had an earlier question about the task force. This updates for you, since the last time we met in July. We had a final recommendation on screening for skin cancer. August we had one on-screen lipid disorders in children and adolescents. September screening for late tuberculosis. In October, by Medicare interventions for breast feeding. We also had some draft recommendations on screening for genital herpes, screening for preeclampsia and screening and interventions for obesity in children and adolescents. I can see their upcoming draft recommendations for public comment related to screening for thyroid Council -- cancer, behavioral [Indiscernible] Cardiovascular Disease, invention [Indiscernible] risk factors, screening provision in children aged -- ages six years to five years and chronic condition for menopausal hormone therapy. Those give you an idea of what the task force is working on. I want to update you also on the work of our evidence-based practice centers since the last NAC meeting. We've done systematic reviews on strategies to the escalated aggressive behavior in psychiatric patients, when on omega-3 fatty acids and cardiovascular disease an updated systematic review, datalink to [Indiscernible] suicide prevention, and patient safety and ambulatory settings. Also methods reports on guidance for conduct and reporting of modeling and simulation studies in the context of health technology assessments. This slide references something Lucy brought up as a question before that we do have a Memorandum of Understanding in progress between AHRQ and [Indiscernible] to do update a previous EPC centers reviews. In doing this, we'll update -- the updates will adhere to rigorous policies and procedures of both AHRQ and [Indiscernible] and were working on procedures consistent with the organizations. The topics will be selected based on the relevance, trends of evidence got time and is. Again, we are doing the same collaboration with [Indiscernible]. Data methods, want to make you aware of a few new reports that are out since we last met. One week from the same data source we used to create our national quality in disparities report, we also create AHRQ State snapshots. This comes from the 25th doing quality and despair to report. This is available that you can find on the Web. For example, if you are from Maine, Massachusetts -- Massachusetts, [Indiscernible] or Minnesota call your living in State where [Indiscernible] top-performing State for health care quality and this is a resource that [Indiscernible] online resource that provides all 50 states from the District of Columbia with insights on more than 250 measures related to health care quality and access to care. We also have a chart book on care report ability. It provides a probation about what the cost is for individuals and how many reports -- report challenges for paying for [Indiscernible] care. The decrease of 21% to 21% between 2011 and what he 15 for people under the age of 65. We also identified in this report that Blacks and Hispanics are more likely to add [Indiscernible] than all of the years when compared to whites. [Indiscernible] I wanted to share with you I had the opportunity of visiting recently with the editorial Board of health affairs and they shared with me that they have had both [Indiscernible] blogs and anticipating new blogs to reinforce the phenomenon that maps, which is of course hosted here at AHRQ, is a number one data source of articles published in health affairs. And I believe there is going to be a blog posted today that will talk more about [Indiscernible] and we will update you on that during the course of the meeting that is posted and provide more useful information to you.
Over getting bumper stickers?
I love that. Yes, absolutely. This gives you a feel for some of the reporting that has come out of MEPS. MEPS has been a fabulous resource because of its panel sampling and ability to see individuals over time. It's been the source of very important reports about what has happened with regard to changes, for example, this is the payer mix or coverage mix for ambulatory coverage visits. You can see we were over to compare over time some of the last six months in 2011 through 13, versus the last six months of 2014. 2014 being the year in which the major provisions led to coverage implemented in the Affordable Care Act. You can see in expansion states, the percentage of uninsured visits, in the pink at the top went down in expansion states but hell quite stable in on expansion states. You could see the distribution of the other payers that, for example, that public and marketplace increased in the expansion states. It explains the drop in uninsured in those expansion states, and this is also reflected in the emergency room visits where, again, uninsured individuals -- those individuals coming to the emergency room uninsured dropped significantly in expansion states over time, whereas it remained quite stable in the non- expansion states. MEPS is a good resource to be able to track policies such as the [Indiscernible]. Also out here, my familiar with the healthcare cost and utilization Project, [Indiscernible] project, which hit a milestone of more than 50,000 disseminated data sets. We hit that in the third quarter of 2016. [Indiscernible] is not only a resource that we use here at AHRQ, but we are able to help researchers gain access to these data for their own research projects. This shows the trendline overtimes in terms of the requests which continue to go up over time. 2016 reflects the first three quarters of 2016, but has been consistent in growing resource for researchers in the field. David, you had a question?
David Ballard?
Thank you, Andy. I had a question about the disparity report. I am sure [Indiscernible] but I wanted to underscore this point about the potential opportunities for AHRQ in the disparities analyses space to help health care systems such as my own better understand disparity reduction opportunities within our own systems. I was at the [Indiscernible] national complication in DC the last couple of days. You may know Ascension for the first time has a systemwide long-term goal related to reducing disparities. It's very specific with very specific measures. My own organization and other organizations are trying to advance are work in this space, and while the State disparities work is great work, and I realize you have limited resources but if you can provide leadership helping healthcare systems such as my own, such as essential and others working in the space as well to help us understand what measures might be valid and, essentially, useful to help us with our healthcare system disparity reduction efforts.
David, thank you for that. It's a perfect segue to where I want to go now. Frankly, where we are going the whole day which is to talk about how we can bring AHRQ's work in better alignment with healthcare systems as an important organizational unit of change and improvement. I'm going to set this up at a high level and then we will start to segue into some of the other speakers. This is absolutely the theme of what we want to talk about today, which is, how do we make are work relevant and useful for Health Systems to become learning Health Systems? This really grows out of a message I have been communicating with staff here that I think to have an impact in a way that AHRQ can have an impact that we need to think a bit about the breadth versus the depth of our work. I do think bringing a bit more depth and focus to our work may be an opportunity for us to enhance our impact. That's to think more about greater integration of our work and how we also can procreate, to work with end-users of our work to make sure that it's as relevant as possible for them. To that and, we have very much -- to that end we have been thinking of the health system as a unified concept in which to do this that as I think as we look around, much of the work we are doing at AHRQ could be aligned in this way, consistent with what you are raising, David. I think that's what we really want to put out to the NAC today to consider along with us and to guide us as we think about this. Many of you are familiar with we're have heard of the term of a learning health system. I put up here he definition that the Institute of medicine, National Academy of Medicine, has been using and its work focused on this area. They say that a learning health care systems one designed to generate and apply the best evidence to the collaborative healthcare choices of each patient and provider to drive the process of discovery of the natural outgrowth of patient care, and to ensure innovation, quality, safety of value in health care. You will recognize many of the words there are very consistent with the mission goals of AHRQ, but really brought into focus, thinking about their connection with the health system. Again, I think there is a very strong alignment there. Why this thing now is a good one for ark? I think there is a few things that are really changing in the healthcare environment that make this quite an urgent and appropriate focus area for AHRQ. I think, first of all, we have of course, the emergence of electronic health records and Information Technology, which is providing a tremendous opportunity in terms of being able to support the effort with a new -- with new sources of great -- new sources of data in greater detail of healthcare experience that can be harvested to generate new knowledge and to be used, in fact, to assimilate and implement changes in healthcare delivery for quality and safety improvement. Also, I think were all aware that health care practice is rapidly changing. It's consolidating and many cases with [Indiscernible] large systems is becoming more prevalent. I think the emergence of the systems in many cases is associated with the growth of infrastructure in those organizations that are focused on trying to generate knowledge, adopt knowledge, and apply knowledge in a more systematically, thoughtful way to improve a personalized healthcare choices and shared decision-making of the patient. I think there is a tremendous opportunity for AHRQ to play a constructive role in this movement, and to understand that. This ties into what many have talked about in evidence generation movement. There has been a lot of activity at the FDA, NIH, and other partners here at HHS have found harvesting some of the knowledge that's being generated by electronic health records as a way to allow us to do research we're quickly, at lower costs and cost so forth. We are quite interested in helping to fuel that, but we are also interested in the other end of it. We think it's particularly important in the history of AHRQ has been tied up with getting important evidence [Indiscernible] two questions identified. If you start at 3:00 on the figure thinking about practices, clinicians, and the patients and families informing what questions are coming up in healthcare delivery? How do we turn that into important research questions that potentially could be answered with new sources of data. Where AHRQ has a special role to play that I think is not as well-developed in other parts of the strategy thus far is, how do we make sure the information is being fed back and applied at the front line of care? Again, our goal is not just to have research where we know what the right answer was that applying it, what to think about how to make sure where the ongoing feedback moves. A lot of our talk today will be about how Ark plays an important role in the cycle of generating evidence and also feeding back evidence to make sure its affecting the front lines of care. The final reminder of how we are going to do this, I have now completed my update with you. We are going to turn when we come back together to talk about the learning health care system, talk about what AHRQ is already doing in this space. And after there is a lunch break. We will then come back and have all of the speakers along with may be available to get into this were deeply with you in the afternoon. Thank you, very much, for the opportunity to update you about AHRQ.
We will have a schedule adjustment in that we are going to take our break now. Because of the time and then we will come back and do implementation if that's okay. We will keep it to a 10-minute break. That would be fantastic. Thank you, very much.
Thank you.
[The event is on a 10 minute break]
[Captioner remains on stand by] .
[Captioners transitioning]
[ Captioner standing by ]
Let's try to come back to order. That was quick. [ Laughter ]. Thank you very much. I do know that since we did introductions, [ Indiscernible ] in credit -- Kevin [ Indiscernible ] have joined us on the phone. Andy, I am going to turn it you for introductions.
Thank you and welcome to the new folks on the phone. What we are going to do is we are going to now shift from my overall update into three specific talks from different members of the senior leadership team here at AHRQ who will give you some insights into some of the work that we already are doing that aligns with this concept of the learning health system. This is really trying to get everyone up to a shared understanding of our work so that we can engage all of you in your feedback and thoughts about how we might pursue this topic if you agree with us that this is a good unifying theme for us. The first of those speakers will be Dave Knutson. Dave directs our center for delivery organization and markets here at AHRQ . [ Indiscernible ] acronym of [ Indiscernible ]. [ Indiscernible ] work is to improve the quality and efficiency [ Indiscernible ] decision-makers on organization, payment delivery, and markets. [ Indiscernible ] research portfolio covers organizational, behavior, delivery system, markets, and external factors. Dave came to us and AHRQ in March 2016 from University of Minnesota where he was a senior research fellow [ Indiscernible ] and management. I will turn over to you, Dave. Thank you very much. I will then introduce the other speakers before their talks.
Great. Thank you, Andy. The comparative health system performance initiative has been presented to you numerous times before. Today I'm going to provide a very brief refresher, and then develop most of the time to a progress update. Through it all, I want us to consider the role of this initiative in achieving AHRQ broader goal of supporting [ Indiscernible ] health system performance improvement. And to sort of set up that consideration, raise a couple of opportunities for this initiative. You will learn that this initiative is addressing a comprehensive array of them very important topics related to attributes of health systems and its relationship to performance. Many of these topics will touch upon what we have come to see as competencies related to learning health systems. Some in-depth like when we do a search and research IT use, but some may be less in-depth. We have the opportunity to build on what we are doing going forward to make sure that at the end of it, we have a comprehensive research agenda related to understanding the attributes of learning health systems and how those attributes contribute to performance. The second idea I would like to raise is while we are working on a broad dissemination strategy for the findings of this five-year project, we also recognize that there may be opportunities to engage the systems we are studying in a more interactive relationship. One that allows us to possibly find out what they think they need to improve to gain insights on the inner workings of the system and easily understood from outside and even identify additional data sources that are not easily available now. And then also to help us interpret our findings and to consider how those findings can be feasibly implemented to improve their capacity learning health systems and also to improve performance.
First, and he mentioned that we are seeing a lot of consolidation particularly in the hospital markets but also with medical groups. This map is produced based on some internal intramural work that we're conducting using various sources of data. You can see that it shows that 34% of medical groups are positions in the country. In 2013, grouped into systems defined in a certain way primarily vertical. There are still quite a bit of variation. My home state of Minnesota, it is virtually impossible to find a provider who is in a part of the system. You can see in other parts of the country that health care is provided predominately by independent practitioners. By way of refresher, the comparative health systems performance initiatives purpose is to identify classify and compare healthcare delivery systems to accelerate the dissemination and implementation of patient centered [ Indiscernible ] evidence. It is primarily related to capacity building and attributes of health systems. Is still surprising after so many years of imagining that prepaid group practices and vertically integrated health systems because they have all the tools at hand to produce high-quality efficient care, we still don't know much about how that really works. I think this initiative really has the opportunity to push the tools forward. As some of you remember and to give you for those who don't a little bit of the basics, it is a five year program funded by patient center outcome research trust fund money. The overall budget is about the DA million dollars. We have awarded cooperative agreements to three centers of excellence. Dartmouth in collaboration with Berkeley, the [ Indiscernible ] center of excellence -- the national Bureau of economic research in collaboration with the health research and education trust. The PI at Harvard is it David Cutler, and then went in collaboration with Penn State and the PI is Cheryl [ Indiscernible ]. It is important to note that each of be centers of excellence has in the proposal proposed a number of specific research projects. Each of those is led by API who is highly regarded in their field. We also awarded the contract to Mathematica policy research in January 2016. They serve as a correlating center to make the work of the centers of excellence with the work that we are doing intramural he and help us put together, synthesize it, add some additional analysis, and produce what will be a compendium of US healthcare systems that I list and the bottom here. That will be offered on AHRQ website. It will be part of our broader dissemination strategy. As I mentioned, AHRQ has been conducting and will conduct intramural research to augment the work of the meeting centers. A little more detail about what each of the centers of excellence are doing. This is where I might make up a little bit of ground in terms of our agenda today.
Each of the centers has different -- when you look at it in aggregate, it is highly come from entry. That was the thinking in selecting these various centers among others. A number of them have like RAND, the national perspective using secondary data such as Medicare performance data, but they also have the opportunity to do some very deep dives with health systems in collaboration with four states, California, Minnesota, Washington, and Wisconsin through collaboratives that already have deep relationships with systems. They are focusing -- each COP will focus in some ways on different populations. There is some overlap. For RAND, the underserved populations especially specific conditions populations with specific conditions to be determined in collaboration with others. Use of health IT is the main focus. This will be an area we are likely to find them tapping into some key competencies related to learning health systems, and we would expect that we should be able to do some good cross watch in that area. They are interested in Medicaid ACO's and Medicaid health homes. Dartmouth also is the national focus primarily in its national focus through a large survey of our nationally representative sample. In addition, no be looking at claims data spanning the US. They also have an opportunity for deep dive through their existing partnership with a high-value healthcare collaborative. There will be focusing on complex patients with multiple chronic conditions. Again, patients with specific conditions or cohorts of certain types that are to be determined in cooperation with [ Indiscernible ]. Will be focusing on HR Q priority populations. In terms of clinical conventions, -- innovations, biomedical, care delivery, and patient engagement. The national Bureau of economic research also has the national focus. It will be primarily relying on secondary data covering a large portion of US. This is a place to introduce the fact that we are very much interested in the characteristics of the markets in which the systems are operating. As you can well imagine especially with larger healthcare systems, they both influence the market and are influenced by demands for the market. We need to characterize the markets in order to understand the phenomenon. So and DER is going to do that I will be building out from that in our own work as well. They also have the opportunity for regional deep dives with four states primarily [ Indiscernible ], Colorado, Massachusetts, Oregon, and Utah. You can see the individual populations they will be looking at initially. Pediatrics, postacute care. And treatment of individuals in multiple settings. For care delivery, they're really interested in looking specifically at cancer services and pediatric services.
I mentioned the compendium. That will be in some ways of the vehicle by which we pull it all together. The overall goal is to foster greater understanding of the characteristics and practices of high-performing healthcare systems and encourage and support for the research leading to improvement. So it is a broad goal. It is a long-term goal for us. The five-year project, it will be updated all the way along. They want to think about it going forward and it is something we offer for the long-term future. In the short term, however, were going to populate the compendium with initially a list of all the US healthcare systems that have been defined using a common denominator definition that we have agreed to among all of us as a starting place recognizing Derman different kinds of systems out there. We need to start someplace. We are going to start by focusing primarily on vertically integrated systems were in some ways, they are very important in and of their own right, but in some ways, it is more feasible to begin to assess and identify them and assess their characteristics. So we will provide that information in the compendium. We will also associate those systems with key organizational aspects primarily structural in the beginning. Facilities by type, size, and composition of medical staff. Hopefully more more as we get into it especially with the surveys, the managerial processes and some of the internal operations. There will be a linkage file associating provider organizations with the systems by name. Again, our long-term objective or to expand existing our role in supporting healthcare systems capacity to become learning healthcare systems to improve performance particularly implementing evidence.
So progress today, we have initiated collaboration among the centers of excellence to produce synergies. Were quite excited about it. We are seeing the spirit of collaboration already. I will give you some examples along that front. We have established and convened our COP workgroups. They are our health systems [ Indiscernible ]. Through these workgroups, we have began harmonizing measures across his difference -- different centers of excellence. We have began procuring data and conducting analysis. That is both within AHRQ, the COE and also the quantity center. Just at the end of September, we can bring -- convened our first in person COE meeting. Our workshop was quite successful. And then immediately after that, we convened our first technical expert panel to be a signing board and included individuals from health centers representing associations. Here is what we think we are going -- if you have any ideas on how we can improve our plans. Some examples of the progress we have been making, first along the collaboration front. We have reached an agreement on an initial common denominator definition of health systems for the compendium. The COE will be using themselves a wide array of different definitions. We needed to get started. A feasible way to get started is with our common denominator which is primarily with integrated systems which is about 750 in the country we can identify. We have reached substantial agreement to harmonize a number of attributes and performance measures that measure similar concepts across the COEs. That will allow [ Indiscernible ] and or cross [ Indiscernible ]. The survey content of one COE was sent to all the others and us to obtain suggestions on information useful to other COEs research and for our compendium. We also have active sharing of the COE which we expect to continue in the future. Among the COEs, with experiences we have had to date validating new data sources, and that has been exceedingly helpful for all of us and helps us all of the learning curve.
I will give you an example of the data sources being used by the COEs. Primary data being collected include national surveys, as I mentioned. Interviews with healthcare systems, also ACO's, physician organizations, and with regional partners that I described earlier. There are 24 sources of secondary data being used or will be used by the COEs across the [ Indiscernible ]. It is amazing. They are scooping just about anything anyone can imagine. They include from provider identification and linkage to systems work, we have been doing with Pecos which is the CMS system that registers positions and all providers related to their IDs to bill Medicare. [ Indiscernible ] which is developed by the [ Indiscernible ] planning and evaluation program which links -- which uses Pecos and Medicare claims data to identify physician practices but also to specialties. Will also be able to obtain from CMS information which providers which alternative payment models. In the near term, we have pioneer [ Indiscernible ] so we can start to look at systems that are in these ATMs and not -- that will be a real productive area for us. System attributes will be -- measures will be built a stone planed data. Other administrative data as well as American Hospital Association surveys. Two examples of many sources. From utilization cost information, claims data I merrily Medicare and Medicaid.
An example of the work on harmonizing measures of system performance for the compendium, I want to walk through all the detail. I think the point is if you look on the left column, we identify measure concepts broadly. Cost, evidence-based care, other best practices, safety, patient reported data. Each of the centers of excellence is come to the table with their own definition. In some cases, the definitions are Tweedledee, Tweedledum, and we can actually agree on a common definition of the cases. We just need to be explicit so we know how to crosswalk and which measures are applicable for which questions. That has been a very productive process so far. We expect that to continue successfully. So next steps. We are creating variables and linking data. We are creating public use files. We are continuing to conduct research projects, and they are expanding. We won't be beginning overtime to disseminate data and findings to research papers, webinars, the website which we have just set up in addition to the compendium. Again, to bring us back to what we should be thinking about with this update is a couple of areas where this initiative can actually serve the broader objective at HR Q as we discussed the today. How can we better support health systems and become learning organizations as a unifying principle were brick that [ Indiscernible ] so much about competencies related to improving quality and efficiency. So the two questions again or the two observations. One is yes, we're covering a lot of ground. We are addressing a comprehensive array of topics. Many of them will touch on what we might consider to be the core competencies of the learning health system. Some of them are in depth but some may be been an week and we have the opportunity to build out and make sure that as we move forward, we are comprehensively addressing the issues of not only the competency areas, the domains, but recognizing the [ Indiscernible ] agreement today about defining some of these competencies or how to measure it. We can really work on filling backspace. The second thing is, as we had a dissemination plan, that is fairly traditional and thoughtful, but how can we really consider how we might engage the system that we are studying in a more interactive relationship. Again to find that what they think they need to improve and gain some insights about the internal workings that we may not have access to otherwise, and maybe some data we may have access to now. And then later, as we generate findings they can help us interpret those findings and also consider how those findings can be made useful for them as they actually try to implement for improvement. So with that, I will just open it up for questions.
Sure. I am going to ask the question I always ask. When I am not here anymore, Shannon, could you just ask this question? You mentioned Medicare and Medicaid. Private employers self-insured data, is that on the table as well? Is there some way to access that?
In that 24 sources, what I would like to be able to tell you is [ Indiscernible ] data or data from private insurers is in there, I believe it is, but I need to assure you. Mike is one of our project leads on this.
[ Indiscernible -- low volume ].
So the 18th [ Indiscernible ] data would be available and I just want to make sure you have an accurate answer to that.
I wanted to add that employers are really anxious to offer high-quality networks. They have hesitated because if they have several of the national [ Indiscernible ], they each have their own definition of quality and how can you explain to your employees that this health system is in one high-quality network, and it is not considered high-quality in another. So if AHRQ could have the measures that say this is high-quality, that would be very helpful. I know employees would be happy to skew the numbers there, direct contracted it is necessary and look for plane design features that could help move them. I know that all of the health systems are sharing their data with the healthcare fraud prevention partnership with CMS and others so there is data out there that hopefully, we can figure out how to tap into.
Those a very good suggestions. Yes, I also have been into managed care [ Indiscernible ] for many years and see the opportunity with self audit employers. We are trying to develop core networks. Thank you.
David Ballard.
Thank you. Great work. And the -- Andy in -- [ Indiscernible ] in the mid to late 80s, my world was simple. I worked at the Mayo Clinic. Today, you can still pretty easily identify the Mayo health system. I think it is important when you do this health care system work, that you understand the complexity of some of these healthcare systems and that you sort of adequately represent that complexity in your analysis. For my or organization, we have 80 joint ventures in healthcare. If you actually want to know about our [ Indiscernible ] performance, we had a joint venture hospital that does [ Indiscernible ] cardiac surgeries a year. We own 51% of that hospital. If you try to characterize our cardiac surgery without looking at that hospital, you would be missing 40% of the valves and about 25% of the bypass surgeries that we do. That organization -- that joint venture -- we manage essentially like we manage our fully owned hospitals. I think it is really important in this health care system work that you have input from people who work in some of these more complex systems to make sure that the healthcare system definition actually represent with these healthcare systems really look like in 2017 and beyond.
Yes. That is a very good point. In some ways, it actually supports the conversation we're having today because we recognize that has we try to identify systems and understand the incentives within them and how the relationships -- without actually getting closer and finding out about some of these complexities, the data that are available to us through usual sources really don't capture that kind of complexity. So a deeper dive, a closer relationship with systems is possibly the best way to get at some of that.
[ Indiscernible -- low volume ].
Simple question. In concept with this figure on positions on systems, to we know what worship of the US population is cared for by positions in healthcare systems now?
I don't. That is a good question. I would like to know that myself. Why don't I go find out and we will get back to you. It is a good question that I don't know the answer to.
Sandy.
I think it is a lot also.
A question I have when ever we collectively [ Indiscernible ] any funding agency develops these large centers or grants is there a plan to make these data available to others so that there can be leveraged by other investigators outside three large consortiums?
Absolutely. Think you for bringing that up. That was an important part that I missed. Part of the companions offering is really, we are thinking about creating data basis that are useful for external researchers. Not only through measures that allow for generalizability, but really help with difficulty in doing organizational research where one off research is so expensive and hard to do. We are trying to maybe cut the learning curve and also the difficulty in ramping up to even do any of this work. So yes, that is actually a major objective for the compendium.
My guess is you are already doing this, but just in case, it might be useful once this is up and running, I know how to difficult these things are to get started and get all the bugs out of them, but to have some periodic interactions with external researchers whom might be people who use this to get some suggestions and feedback about how this stuff might be doing overtime. Some sort of advisory board or AHRQ funded working conference with different people to come in and look at what is being done and react to it .
Thank you. That is a great suggestion. I just want to add that some of the things we would love input on later today when we discuss more in the panel is also the challenges of on the one hand trying to engage and get help systems to share information with us, and then whether that will create barriers if we say well, we're going to release some of this publicly, and how do we walk that line in some ways? That is a common issue that comes up in a lot of research. I think it is also part of why our evolving strategy is being referred to -- about trying to co-create with the healthcare systems and figuring out what the value is that they would get from this information and that would lower concerns about how the data may be shared and how to also think about what aspects of the data may need to be confidential versus what can be shared. I think there is a lot to be thought of in that regard. That would be very helpful to us both in terms of your own knowledge in this but also general strategies of how we might further pursue this any way that meets the health system needs as well as the research [ Indiscernible ] needs.
That is why they give you that 10 figure salary. [ Laughter ].
I will just say along those lines I think will come back to this. That is work that [ Indiscernible ] has been doing intensively with health systems particularly around not only what motivates them to participate, but a lot of the data sharing questions. I think we will have a chance to talk later about what that collaboration provided in terms of opportunities to really both reality test a lot of this and then kind of learn from what we learned through those efforts.
For the public record, the AHRQ director does not make a 10 figure salary.
[ Laughter ].
How many of those numbers went to the right of that decimal point?
[ Laughter ]. [ Indiscernible -- multiple speakers ]. Don Goldman had a question two.
Thank you very much. This is really great stuff, and I appreciate hearing about it. I skipped ahead and look at Arlene's flight as well. I am going to make a comment that is maybe a framing comment that people can address when the time comes. When we talk about learning healthcare systems and collaborative and these kinds of terms which I see on the slides, I am reminded of the work that I have been involved in with returning health systems. The terminology is really problematic and is interpreted by different people in different ways. Almost every request we get from CMI says maybe [ Indiscernible ] would be a good place to help set up a learning system or a collaborative comic and it is usually cut-and-paste from some document. As I listened to this presentation and also look ahead at what Arlene is going to say, there is a huge emphasis on learning from data and beating back evidence generated by increasing [ Indiscernible ] data -- clinical data back into the workforce so that they can practice evidence-based care. That is really important, but it is not what I usually see in the request to set up learning systems and collaboratives, which are generally designed to bring systems either macro or micro systems together to learn Realtime about their innovations and their testing of innovations, the development of prototypes, and their deployment. You may or may not know that we are involved in putting together a playbook for five foundations that will help people take care of folks who are patients with complex health care needs and cost a lot of money. We can put together that playbook, but the real trick is going to be implementation and learning the real [ Indiscernible ] limitation. So I am wondering if we talk about this learning healthcare system, if we can also look through that lens of how the systems will learn together about were innovations are working and where they are not and how they can be adapted in scale as opposed to only or mainly looking at what we can learn from the large troves of clinical data using modern analytics and dissemination of those findings.
Thank you for the comment. It is so consistent with what we are thinking about here at AHRQ. I think is very helpful to hear your point. I do think that just to emphasize, something I try to say in my update, I do think that many of the partners in HHS are thinking about what you just mentioned which are useful ways to extract data to make research more rapid and less costly. I think AHRQ special role is about both generating questions that come from frontline practitioners and that be back because I do think that is something that is a little bit less well attended to, but it is critical if we are actually going to get to evidence-based practice. So that is a perfect segue to our next speaker, but something that we should continue to talk about as a strategy of how we make sure we are doing that. So let me just briefly introduce my next speaker who is someone you have heard from before and that is Dr. Arlene Beerman. She has been at AHRQ since August of last year so a little over a year now. She is direct -- director of our Senator. for evidence and practice improvement which has five divisions. You probably heard her talk about some of these before. That includes the evidence-based practice centers, the was prevention services task force, the division of science and patient engagement, the division of health IT, and division of practice improvement as well as the national Center for excellence and primary care research. Arlene herself is a general internist and a geriatrician and health services researcher. Her work is focused on improving access quality and outcomes of health care for older adults with chronic illness and disadvantaged. Arlene, thank you for helping to get us all up to speed about some of the ways [ Indiscernible ] with learning health systems.
Thank you, Don for setting me up because that is exactly where I was going with this presentation. I am going to try to be brief, and then we can come back later if you want more detail. Some of the stuff I presented, I presented here before. Basically, what I want to do is talk about some of AHRQ experience with doing learning health networks and what we have gone from that. And then talk a little bit about getting more granular down to the point of care and care delivery is what do we do to apply the best evidence, see evidence of limitation at the point of care within a practice or a system, and then also how do we generate evidence about what works both clinically, we know a lot of triers -- trials don't address all populations but also -- evidence for improvement. Artificially, I broke my talk into the evidence of limitation, but we all know we want to see. We are all continually learning ourselves from the process.
This is a slide taking from ONC. I think the opportunity here is both the growth of help systems and growth of the availability of data. The use of data from personal health records, electronic health records, health information exchanges, and going up to a larger system national and [ Indiscernible ] analects bringing in a lot of sources of data, and once the data is available, it can be used to inform and learn on multiple levels. Patient level, position level, practice level, and really population management. How do we move from individual care to really improving health to populations as well as public health. So this is a slide where I just pulled three examples of AHRQ prior work in supporting learning committee communities. One was the Medicaid medical directors learning network. It had a number of successes in improving care and Medicaid including antipsychotic medication use and children, hospital admissions/readmissions work and some work around early elective deliveries. Our innovations exchange sponsored a number of learning communities. Actually it has had some successes on a small scale. There was an intervention to improve birth weight, reduce low birth weight in Ohio. That involved your support that is actually -- the state of Ohio is generalizing that in implementing its statewide. We had just completed a learning community in Florida to increase patient and family engagement and [ Indiscernible ] and hospitals that the state of Florida is now going to spread across the state. And also, a really interesting learning community in Detroit, which paired paramedics and first responders with the help system and vulnerable people who were frequent users of EMS to get the support they need and keep them in the community and reduce their emergency -- in Detroit, the city is now supporting the continuation of that. So I think there is a real opportunity to kind of go back and look at the learning committees we have sponsored in the past in a whole host of areas and see what worked and what didn't and what can we learn about the effectiveness of learning communities and how to make them more effective in the future.
I know you have all heard a lot about evidence now which is our state and regional collaboratives to reduce cardiovascular risk. David Myers presented here on that. Basically, I think the nice thing about evidence now is it is really a model of integrating evidence implementation and evidence generation. We are trying to improve guidelines and [ Indiscernible ] care and at the same time learn what works in terms of practice facilitation in small and medium-sized primary care practices to improve care so that could be generalizable in terms of primary care transformation. Another successful collaborative we had was the EDM form which we funded since 2010 through a cooperative agreement with Academy health. Our funding is coming to an end, but many of the things that were developed through this are going to be self-sustaining. Really this promoted a collaboration among researchers, clinical operations, clinical informatics, and others to really accelerate the use of data to inform learning health systems. It has had a number of notable successes including a committee of practice where people shared ways of using data to inform health systems improvement in various systems. I am going to switch now into what does AHRQ produce both in terms of tools for implementation and evidence generation. I would like to start with a hypothetical example just to get something concrete to anchor it. So if we have a learning health system that wanted to improve stroke outcomes and reduce costs in ACO population that they were serving, one of the things we like to discuss is who should be the target people involved for AHRQ and thinking about the chief medical officer, the chief informatics officer, ice presidents, quality and nursing -- they have goals on taking a population health approach -- goals on a couple of different levels that I think are really important. One is just improve health outcomes in [ Indiscernible ] patients with stroke. There are a lot of evidence-based guidelines. There is a lot of howling -- we know about how to improve stroke outcomes. But what about people with TIAs or people in the community who are not presenting with early systems in which a stroke be prevented. How do you reach out to those to reduce stroke incidence. And in a broader more long-term population, we know that many of the causes or risk factors for stroke are modifiable. So what are the needs of the help system in order to do this? In terms of evidence implementation at the point of care, systematic reviews, guidelines, quality indicators, E measures, tools for improvement, registry. What are all the different mechanisms the system could use to improve their delivery of evidence-based care? But then evidence generation and really need to know what works and what setting for which populations on the effectiveness of interventions and rapid cycle learning and feedback. Beth, your commentary at that was in the gym internal medicine last week really pointed that out that it is not enough just to hide the evidence and the tools and the measures in the reporting, that you really need to combine it with some sort of structure intervention and feedback to actually have an impact on outcomes. So how do we move from all these tools we have to actually something that supports system change and ultimately improve clinical care in population health.
I'm back to the framework. I'm just going to go over some of the tools we have for actually lamenting evidence at the point of care. I've spoken to you before about evidence-based practice Center program. We actually have gotten a lot of good feedback from you which we are starting to implement. I think the real question is you need the evidence in order to implement it. This systematic reviews are the cornerstone for that. How do we work with systems to identify [ Indiscernible ] and questions. Both clinical questions and organizational questions about models of care. How do we increase the relevant and the impact and how do we increase maybe a pool rather than a pushback for our evidence reviews. We just have a whole host of tools that help people find and implement the evidence including the national guideline clearinghouse, the national quality measures clearinghouse. One of the things we haven't be about internally is can we also improve -- integrated digital strategy and figure out ways to integrate our tools that you can come to our website and get the systematic review, you could get the guideline. You could get the quality report. You could get some tools for improvement. We are thinking about ways of how we can effectively align our resources to better support the information needs and the tools and make our tools more available and accessible. I think one of the exciting things that was released in the last few months over the summer was the AHRQ integration playbook. If you haven't taken a look at it, I would encourage you to go online because it is really a fully interactive online how-to guide for integrating behavioral health and primary care. A practice or a system could actually go on there and see their readiness. It could guide them through how to start depending where they are on the process. It assist users in planning for integration, preparing the infrastructure, establishing protocols and clinical workflows, developing processes for tracking patients, monitoring outcomes, and maintain engagement. I would just encourage those of you who haven't seen it to take a look at that. Also how do we build with the work we are doing internally. Also recently, we released a technical brief and and evidence review on patient safety and ambulatory settings. So and follow-up for that, we actually have several efforts now to develop tools based on promising safety practices and and Latorre care settings that were identified by the report that you can see cover a host of different types of information from improving treatment and diagnosis with pneumonia at the point of care to improving the efficiency and effectiveness of information exchange and also ways to collect and communicate blood test results. So we are hopeful that this initiative will provide some new very useful tools for ambulatory safety. This is a new initiative.
It is an example of work across the continuum. The Catholic health initiatives which is a large health system across 19 states, 104 hospitals, is using a decision aid and materials that we have developed for a whole campaign around lung cancer screening. I think this is interesting because it went from a systematic review to US preventive services task force recommendations to a decision tool. I know CMS is also using our decision tool, but CMS has said that in order to get reimbursed, shared decision-making has to happen. It is actually a nice little workable handout that really informs patients on the risks and benefits of undergoing lung cancer screening. So now I'm going to talk about some of our work replete around evidence generation. I think this is really exciting. Our work on clinical decision support which is actually funded through the [ Indiscernible ] trust fund. It was mandated through the affordable care act. We have recently funded a learning network to really bring different stakeholders and providers together in terms of how to learn about using clinical decision support. There are a number of funding opportunities for scale and developing clinical decision support. We have a contract with [ Indiscernible ] to develop an online prototype for a repository so the clinical decision support can be in the public domain and easily shared across systems. I put it here under evidence generation even though some of these are tools is because part of this is really to learn how to do it right. We had clinical decision support available for decades with very mixed results. Part of developing this is to really learn what works in terms of doing effective clinical this is -- decision-support and doing it at the point of care. From a paper by [ Indiscernible ] comp but there are five breaks for clinical decision support that haven't been widely applied in the development. There is a variety of information. What is the evidence to the right person. Is it going to the nurse, the position, the right intervention? Is it useful the way it comes up. The right channel, it is it in another IT system? And also really thinking about workflow so that it is not interference, but is actually supporting the care of patients. We are hoping with [ Indiscernible ] were, we are going to be working with learning and developing in a way that is more effective.
The first time right after I started, I presented around our H IT portfolio, and I got great device. As a result of that, we really focused on patient reported outcomes and patient generated data into be HR is because we think that is key data that is missing. Having backdated there will allow us to learn a lot about what is effective in managing different patients. Especially more complex patients. Basically, if this is the same theme that you collect the same data, and is useful for multiple people in multiple purposes. It is useful for patients and clinicians and clinical management shared decision [ Indiscernible ], hospital leaders and clinicians in terms of managing the system. What are their outcomes? I know CMS is doing this more for hip and knee replacements. What are the outcomes? Insurers, hospitals, leaders, and also for research. So collecting the data and having it in a way that is standardized and useful and comparable across systems I think would be a big advantage. So we have actually made a lot of progress in the last year in advancing this focus on integrating patient reported outcomes into the electronic data to support patient engagement and patient centered care. So we actually did a virtual technical expert panel through our action network which was really helpful. We learned about work being done across the country, different health systems and where they are in this. We have and EDM for a meeting where we brought together a whole range of stakeholders including health systems, developers, patients, researchers, and got a real sense of what the needs are and what the state of AHRQ is. We have a special emphasis notice out. We are looking for research applications on how to effectively incorporate [ Indiscernible ] in THR's. Excitingly, we have just received an award from the [ Indiscernible ] trust fund -- we are partnering with ONC, AHRQ and ONC are leading this. Also NIA, NCI and [ Indiscernible ] are also involved. Our goal is to really developed the standards and prototypes for getting functional status measures for physical punching into THR's and then test them within health systems. I think that is exciting work that could really advance the field and really give us the data we need to generate evidence. And then finally, hot off the press, and he mentioned a couple of notices of intent. We put out one yesterday on we are looking for applications to scale effective uses of patient reported outcomes. We know there has been examples across the country in different settings. Can we take the systems that have been [ Captioners transitioning ]using those well and extend those across to other systems.
There are providers at work in isolated conditions. There's financial constraints and small practices to do some of this work. Primary care has really been beset by lots of demands and has huge needs. There was a quote this was someone that visited AHRQ and we are trying to make it through the day. That's how primary care practices are feeling. So what can we do with this to really release -- relief burned. The other thing is with this expansion of a seals is it is back to the old days where big systems -- not giving them the support they need to do what they need to do so how can we make sure that the vertical systems and provide the support. AHRQ does have some resources that we think could be repurposed or support some of this effort. We funded the practice-based research network for years. This was a little older. There's 179 ResearchWorks across 29,000 practices over 150,000 clinicians . This is a huge resource. Maybe there is creative ways in tapping into this resource. One thing that they did was -- it was a crop live -- collaborative for patient management. Just to think about I love ideas on how we can use these to support this role. The other thing is project [ Indiscernible ] you heard about. It is initial funding for something that is taken off. To think about how those kind of mechanisms can support more broadly and improvement development across different practices and we are using it now for disease specific there is some models for complex patients with multi-morbidity.
I'm going to stop there and I'm looking forward to hearing your ideas. Thank you.
Thank you, Arlene. Just getting back to the issue on soliciting input on topics for evidence entities. Of wondering in addition to sort of reaching out getting that input from the field is there ways that we can proactively [ Indiscernible ] by monitoring changes in policy, new payments in penalty areas. Of thinking about the panel legislation that was recently passed. I know my own delivery system we have been designated entity in that. So we are searching for an evidence -- we will need to do ourselves. That was one example of the monitoring these changes so we can start to get the work done in a timely response.
Lucy, that is a great example. Is there a way that we can get that information to you to know that this would be a useful area for us to do it and work with you in trying to develop questions and seeing the feasibility of doing an evidence. What are the questions out there that we can do evidence that might be more timely or have a greater impact are more relevant to users.
[ Indiscernible - low volume ]
Mary?
I finally figured out how to connect.
Thank you both. I am very excited about the movement and orientation toward AHRQ and the learning how the system and the way of thinking about advancing the mission and vision. I had the great fortune to sit in the meeting. I see that it is a terrific framework. Just a couple of thoughts in this has to do with a way in which your envisioning the learning health system, evidence generation and implementation. I know this was an out of patient net. I think this is a great opportunity to go beyond the way in which we currently think about learning health systems and the network not just the help system traditionally. In thinking about evidence are referring to the diagram that you have here that has that purple evidence generation. To think about the kind of science from which these systems biomedical research is extraordinarily important, but as we think about the increasingly important intersection of health and social needs drawing much more from the social science is really powerful. I'm here to focus on individual patients which is central. As we move forward in learning health systems, the attention that will be paying to other groups family care grippers or population is also going to be important. I think placing attention on a much broader group or subgroups of people from whom we can learn and then produce the kind of evidence that impacts a much broader group. I think early work thinking about the clinicians very much need to be brought in to think about the team broadly defined not just how the FEIS -- professionals. Arlene, I think this notion across the continuum is essential. We been doing some work with learning communities and it's unclear to me about whether the lessons learned in primary care to care -- I think we have to test them out across the continuum and focus on the post acute. There might be some lessons from the projects that are focusing on building, learning communities, home health, etc. So just a few thoughts as your work progresses. I'm really excited about this orientation and this approach and this partnership.
Sandy?
I want to follow up a little bit on what Mary said in taking a slightly different direction. I'm also very excited about this program because it is a role need. For my perspective, when I was practicing actively and when I was on [ Indiscernible ] task force or Medicare advisory groups what I found the biggest deficiency was not evidence for disease -- there's a lot to learn about both of them and I think they are both very important. There's a real dearth of evidence. What we have is continually between data and information and understanding and evidence. I would like to see as much possible on the true evidence generation. What we see is to the collaboration networks is a gift to the issue that was raised before about speed and the need for information versus the rigor of the information. I think AHRQ is situated to help inform that intersection . It is and uncomfortable place to be because nobody is ever going to like you from either side. [ Laughter ] from the evidence synthesis perspective what is the right information that we need and how can we improve the rigor of the data and at the same time to it without disrupting the functioning of the system. I have ideas about everybody on the table. Nobody knows how to do it. Every time I do a trial, -- it so logical but how do you do it is a very different thing. I think that's an area that AHRQ -- because it has grounding in both areas can try to -- while it is doing it advance. I want is the emphasis on how can we come up with practical ways on improving the quality of data wall were doing these other things.
Just to say that you had me at difficult and thankless. [ Laughter ]
Arlene, I was thinking with your hypothetical example of stroke about going in the total opposite direction. The need to actually have systems work well between primary care and subspecialties because stroke the biggest intervention has been using Euro interventionalists in the AD -- ED. I wouldn't want that perspective to be overlooked in this broader effort.
That is why I wanted to make sure that there is that continuum and not to lose that out like reducing popularity and reducing stroke. The big ticket things gets the biggest attention and how do we make sure that we are inclusive and get all the different levels and do the things that you are talking about. I would welcome how we could -- as we move forward, that is a danger that we want to avoid.
Kevin?
Hello, everybody. I want to make a comment. Thinking about learning health system research I see two components. One is what is the system and two what is a science for learning health system research and application? Arlene, I was glad to see you mention things like practice-based research. I think Dave's presentation -- I think it's important to look at the systems. Those are obviously critical but I think we need to think of systems -- health systems that are a bit more loosely defined. I think networks of primary care practices and caregiver networks and make sure we are not exclusively thinking about vertically integrated structures but how a whole host of folks can start to collaborate. Then it is a science, which I think [ Indiscernible - low volume ] research and how it applies in the settings. I would say providing groups to come together to think of themselves as learning systems and then to help the technical systems and from the original research that using this space that is emerging science of applying learning health system research.
Charlie?
Just trying to tie everything from the presentation. Relative to making data available for policy-based decisions and having a safe place for that take place and for Congress recognition of AHRQ being involved in this. I may have mentioned this last time but there is commission that was set up by speaker Ryan and Senator Murray it's going to have its third meeting Brookings on Friday and it's all about evidence-based policy decisions and how legislation might be changed to make that happen. Also I think they are interested in agencies who would be interested in working in that environment. That's what you all are about. I think it might be good to be involved in that. There are some folks that former [ Indiscernible ] secretaries on that commission and several others that you might be able to get they're here if you haven't already.
Jose?
Thank you for your presentation both David and Arlene. I have a question. I guess you can say in the economics assumption is that people go around maximizing utility within a budget constraint. Of course, that's not the way real people work. Now they're even been people that have one Nobel prizes for challenging that paradigm. I think the understanding that that paradigm has problems is what sort of fueled the rage with social psychology and behavioral economics and keys to behavior change in recent years. It seems to me that the equivalent of economics in this context is the assumption and I know I'm not suggesting you're making it but the implicit assumption that if you have the right information and you get it to people at the right time then things will be done correctly. I wonder to what extent the entire effort of AHRQ is focusing on the next step, which is how do you change the behavior of the agents because behavior has to change? What are the institutional features? What is a drawing on social psychology to push the right buttons? It is a question to what extent are those things being considered because my guess is they are really important.
Absolutely.
Jen?
I think might, under scores what clothes they just said. That is we need evidence on how best to implement the evidence. So generating evidence on implementing evidence that is been generated is very important.
Exactly. How do we do that? I agree. That was the best commentary that what we've been doing today doesn't work and we need new approaches to learn what does work.
Naomi?
Thank you. Charlie, -- told me I have to do this. I figured out how to raise my hand. [ Laughter ] very high-tech methods. I just wanted to say that they do so much for the wonderful presentation on learning health care systems. That's what the VA is currently doing right now as part of the commission of care they have been utilizing or trying to utilize it and make dramatic healthcare transformation using this approach. The question is how you do it? That is the nitty-gritty of it all. I have a practical question or practical thought or suggestion. One thing that I've been doing in my new role is visiting all these VA medical centers. There is -- I went to one and we were trying to make some changes and try to expedite the implementation of evidence-based research because that's what health service research does. The implementation is so sporadic and very variable across the medical centers and when I talked to the medical director at the VA I visited yesterday, he said all of these evidence base their bees and interventions are wonderful, but they cost money. He pointed out for hundred thousand dollars or $500,000 is what it's going to take to bring in the new staff, the training, etc. the new staff, the training, etc. So one suggestion and we are trying to look at this at all angles as well -- there are practical ways or if you can do in evidence emphasis of business cases, models where we can use it to convince medical directors or health plan executives who may not be aware of the day-to-day activities and the challenges that providers have to go through, but they have to make decisions on a more budgetary fiscal basis.
I also went to a site visit and several ready projects for implementation that require exactly that analysis in order to get budgets moved so that work is very strongly evidence base can be taken to the next step.
Arlene, I admit that much of what you talked about in the words are not in my paradigm. Learning networks and all the research but what employers are doing is so much like what you are talking about here. The right patient, the right time, trying to personalize thing and user data to help their employees use evidence-based care, helping them to change behavior. I want to make sure that employers are looked at as a stakeholder. They have the leverage in the plan design and the eyes and ears at the worksite to help them get the changes that they need. For example, we've been working with some partners and hopefully CMS will come along also on implementing transparency at the point E-prescribing for doctors with prescription data and providing cost and also alternatives and maybe even evidence surrounding that and what needs prior authorization or what needs alerts. We could use some help -- I'm calling them pitchfork and lanterns issues where we can push together. A lot of what you're talking about here -- we can look at employers as natural experiments on a rapid cycle because they have six or eight months to figure out what they will do the next year, they are looking at ACO's. So let other people figure it out. If we could use the data here to help employers accelerate the adoption of what has to be better, Bennett fractured situation to move into some sort of integrated system that would be great. In addition, we are concerned about the consolidation and the fact that independent doctors are being swallowed up. Talking yesterday at ASCO and the oncologist that are getting swallowed up and cannot function on their own and the cost is so much higher when they are purchased. All of those things are on the table for employers. I would love to keep them engaged.
Final question and comment, Paul?
Just reflecting I think Arlene's comment about vertically integrate systems are trying to form ACO's and not supporting them . The question about how do we put the evidence to make the delivery work? The problem is I think we have the evidence and we don't know how to integrate it. If I had a nickel for every time someone has come to me and said you need your primary care doctor to do this screening on intimate partner violence or on handguns or -- if I put together everything that the evidence is my primary care doctor should do their visits would be 24 hours long. It's my -- part of the answer ties back to Mary's comment talking about the team and not the provider. We have so much evidence and we don't know how to effectively deliver it even in integrated systems were my old job is figuring that out.
I just want to comment on that because I was in a meeting yesterday on developing a research agenda for patient priority centered care because it is the same thing. We need to figure out priorities and also most people of multiple chronic conditions and if you apply all the guidelines, you cause harm because you will have drug interactions. How do we start getting evidence on more complex populations and how do we get evidence on targeting the right intervention and prioritizing so we can have impact? I think those are huge questions.
I want to take this to make a transition because I think José, you raised it and I think many people logged onto this notion of knowledge is going to be may be necessary but not sufficient is a way to bring about the change. I think we could not agree more. We do see learning health systems as important intermediary entities that are grappling as you described, Paul, with all the things that we need to implement. How do we support the activities that they are taking on? To that end, one strategy that we will talk about now. -- I'm going to have him come up who oversees a bunch of activities here at AHRQ including our training activities and really think about whether aligning some of our training activities with this focus on learning health systems might also provide the development of a workforce that is in a position to start to support some of the knowledge transfer and other aspects of what may be needs to be changed in organizations to make some of what we are talking about here possible. So Francis, thank you for coming and sharing a little bit about what we are currently doing in training in how we might need about orienting this differently in the context of the learning health system.
Thank you, Andy. Good morning. I was introduced to her AHRQ before I was getting ready to get on the promotion Truan Mehl. Might to permit your suggests that I come and meet this agency and the goal was to get a career development award. [ Laughter ] this is -- [ Laughter ] it was in the twilight of my year. So I came to AHRQ and it does not have awards. So 20 years later hearing am. I want to talk to today about a new initiative that we are planning which is focus on training learning health system researchers. I want to provide a little context about what we are currently doing in our training and career development programs. I will give you data on grant funding success within those programs and during Andy's presentation give you a sense of how we do programs through oversight with some of valuation activities.
So we are mindful in our development program that stands the time from high school to undergraduate and three training and even into early midcareer and Le Claire transitions within academic researchers. Our focus is on three specific areas. We find pre-doctoral training by finding grant applications as well as by finding pre-doctoral trainee's international research service award institution of training program. We do postdoctoral training also within the institutional program and funding awards and then we find career development mostly focused on her development of clinician and research scientists.
As a result of the Affordable Care Act and the patient centered outcomes trust fund, we have initiated a number of training programs to spend that continuing and fill in gaps within our training program but they are focus on growing patient centered outcomes research. We find institutional training and the first program that you see there is a one focus on building if a structure using AHRQ . Everyone is aware that there is a awful that soup that goes along with every program. The focus is on building capacity on institutions that are research intensive. In addition we find centers of excellent using the K-12 mechanism. We find it a really neat program which is called a pathway to independence award. Here this award is meant to really fill a need that happens in the grid development arena, where the grantees have five years of protected time. They are in a nice nurtured cocoon. Their time is set aside and her salary is supported for five years and then they describe a cliff in which they have to struggle to get their independent award. This award guarantees promotion and research position for successful career development grantees. We also find it some midcareer and senior investigator transitions award to allow for researchers were currently in their research workforce to shift gears and the area of AHRQ research and find it some traditional core development awards using the mechanism.
There are three inputs into our training and crew development activity. I mentioned the national research program. That program awarded $8 million in training grants and similarly I mentioned the PCOR trust fund. There were about $700,000 in new dollars in that program and those work for exclusively for new career to Vollman awards and the other programs were foundational. Those other programs I described -- are appropriated portfolio funding those funds -- we find it almost explain dollars to fund current development and dislocation awards.
Just to give you a sense looking at this graph the funding rate has been at the 36% rate. Around 40% the past two years . This is a piece of information that gathers greater attention and folks are trying to consider the decision between health services research, career development award or one for clinician researchers they be focused on what of the other institutes. Similarly in our program -- this program -- we appointed more than 1100 research projects almost every single one of them has produced a publication. I will talk about some of valuation that we did with this program in a little bit. Success rate hear about 29% over the past three years. Applications are dipping a little bit. Not sure why but we have seen a downward trend in the number of applications that we are receiving.
As I mentioned to do some program evaluation activity both to know where the gaps are in training and crew development but also to determine where the outputs and outcomes for some of our training programs. We funded this year and completed an evaluation of our crew development program we looked at the program from 2013 and focus on grantees who have been had their award completed by 2009. We use three inputs for this project. One was constructing a database from our -- the information for funded grants, we also use AHRQ grants to track publications I we did searches and we funded survey among the eligible trainees. I am pleased to say that the response was 76% to the survey. So looking at it as a perspective to those that responded to the survey, they said that the AHRQ award was highly rated . As critical to their research career, progression, and success the grantees got the greatest training games around skill development and that's what the program is intended to do. Funded research projects were overwhelmingly reflected in our core priorities and 86% of those who responded to the survey said that they plan to continue in a research career seeking research funding through grants. From a standpoint of outputs, the funded care mentees different from unfunded care award grantees in two import ways. It relates to the success rate for getting grant awards. There is 3-1 difference between a career funded award being able to go forward and get future grant awards. Both of them were federal awards. 38% of the grid development awardees went on to get a first award and that was compared to 13% of unfunded grantees. There was an important issue again raised here by the grantees in their survey and that was the transition from protected time into a independent award was won and they did make comments to us about the level of funding in terms of salary support. Lastly, a challenge that we face in both of these programs is being able to find part-time training. That is important for trainees who are in the family building time of their career. Those awards were not able to find part-time work so they have to be full-time equivalent in order to get occur development award.
Similarly in the program we looked at data from 2000 through 2010. 54 % of the grantees were employed in academic and went on to employment in academic setting while 40% were other employment settings. It was not intuitive for us until we thought about it and the training method that research is really foundational for employment purposes rather than future academic or non-academic research careers. While they are moving on to do research, that research is often in association with their employment setting in the employment setting is not always an academic setting. It is interesting to see more than half the grantees continue to -- I am pleased to say that in the first quarter of 2017 we plan to release the next competition funding announcement for the national research service award training program. We are also going to in the process of developing competition award to try to assist that transition to independence for our crew development awardees by publishing a limited competition small research grant award. This is a two year up to $100,000 award to allow them to do some work potentially to lead to their first independent award. We are excited about app you need to pivotal a little bit in our program.
Can you just clarify what that K-R03 program is?
Sure. Throughout HHS there are these limited competition awards that are eligible to career development awardees current in their third or fourth year. So while in the process of completing their training, they can apply a get this award and maintain the K and hopefully take that into a R01. those people are more successful, which is critical. This initiative started in June of this year to construct a set of core competencies like what David was talking about in terms of competencies for systems. Your we are talking about competencies for researchers that would work within systems to develop, implication, and evaluation of training programs for learning health systems. The goal is to train and embed researchers within learning health systems and who can lead efforts to generate and use evidence for improvement and innovation. We discovered while were in this project that compared to classical health services research training, the outputs time and drivers will be much different for researchers functioning in systems.
Move towards a learning health system we say -- conceptual approach for integrating help services research, and as I mentioned, currently our training programs are designed to produce new knowledge. These researchers are with the health centers and the goal in addition to promotion is a production of new knowledge. What we've done in this project so far is sponsor a technical expert panel, which is trying from help services research and learning health system fields to develop a framework inside of competencies for training within learning health systems. The new copies this will build on foundational competencies and help services and patient center outcome research competencies, but address the unique training needs of investigators leading efforts within learning health systems and obviously with the gold towards transforming healthcare. Many thanks to Lucy and Kevin who participants on this technical expert panel. Unlike most projects within the government ideas and AHRQ , it is usually 12 months of this is a six months project. With three meetings and driving towards a completion in December. The folks have done a tremendous job engaging us with their feedback. Where grateful or participation from VA and from NIH. Within this project, I want to define or share with you how we've operationalize the definition of learning research. Professionals who conduct research in healthcare settings who generates new evidence and that health systems can rapidly implement to improve quality of care and patient outcomes. It's help identify 10 domains, which you see here. They are hard at work now driving towards consensus to develop those core competencies that fall within each of those domains. Ones that work is completed, we are planning to event embed those competencies within a multitier institutional training program looking at the K-12 mechanism, which is similar to the centers of excellence that we funded before using that mechanism. A few unique attributes of this particular training program. We are looking for applicants to the program to demonstrate some commitment to ongoing embedding and training and research using the competencies that we have developed. Where expected applicants to demonstrate learning opportunities provided with academic and learning health system settings allowing for the application a mastery of the knowledge and skills outlined across the core competencies that we are developing. One Nye component of the K-12 program is you can support the time of mentors as well as a trainees and so we are looking to do that for both researchers as well as systems and operations leaders. We are expecting scholars to focus on meaningful questions and expanding the system's ability to systematically generate, adopt, and apply new evidence and support of management and personalized healthcare as an example of focus for one of these new training programs.
I will and their. Looking for to your questions especially about the presentation today. One thing that is, as we begun this work is a recognition that while our competencies are focused on individual researchers within systems, we certainly recognize that research and systems is going to have to be a team approach. This project is not taken into consideration the breath of the healthcare team, we are interested in finding out how we might think more about the competencies within a system that would be mindful of teams that would be working with within systems. With that, I'm looking for to your questions.
Paul?
I think this is really cool and I'm supportive of it. I don't mean to sound contrarian, but the K-12 program findings when you talked about them talking about the success rates and how much more successful than the non-awardees, at least we think but that's probably because you awarded to more high-value people in the first place. So I'm sure your thoughts about it because I could also make a hypothesis by awarding these high-value people these grants to make it easier, you stifled innovation and fewer of them -- they had fewer publications -- we don't know but I'm sure you have some thoughts on what a difference this has made not just comparing to a different group.
We should randomize awards rather than -- ink of the efficiencies --
That's what you tell us to do in clinical medicine all the time. [ Laughter ]. That is a really important comment. I will share two observations. Whenever first hypotheses was that we were finding the best trainees who would go want to do research without this award. That is the first thing. The second thing was compare the outcomes for applicants at the cusp of funding. Not the best and not the worst but those that were at the cost. So the cusp for AHRQ is a 30% percentile . We were not able to do that project. The project is a project we were still try to figure out what to do. It became very difficult. Your porn -- point is well taken. What I wanted to get out of this is a better understanding of some of the challenges of getting an award and completing it. We do recognize that that is a significant point.
Lucy?
Thank you, Francis . This is great work. Going back to discussion points of from Dave's presentation they were talking about creating public use data files. I'm wondering the extent in which you thought about leveraging some of that work to create training data sets that both we could use in the -- as we are training people and then also some of the other experience learning opportunities.
Thank you, Lucy. I will make a general comment. One thing that AHRQ has published is the plan that covers research data and access to reach her data as well as the publications. We have a policy and the publication axis is in place. We discovered a consultation with our legal council colleagues that in order to make AHRQ fund it data broadly available and to acquire access to those data that not only AHRQ but components of HHS. So I am also pleased to say that soon I think by the end of this year that migratory authority will be in place so that programs and grants like we heard this morning as well as data generated on training grant can be made available to the public. We thought about at least three ways of doing that. One would be to sponsor data repository. So we had some discussions about doing that. Another would be to use data repositories that exist already and have grantees and training organizations similar data their. The third would be to allow the institution that was funded to get the grant to maintain those -- data in their own repositories and make the data available. So probably make your next are AHRQ will publish a policy on the requirement for public access data generated in research grants. Part of that process is to do a request for information from the public. To get more information on not only how useful this would be but what component of research data files would actually be useful. We've had some general conversations in our training programs. These are the institutional training programs about how we might make these available to the public. I think having a policy will give us leverage to do that moving forward. Would expect that to be a component of the program that we will announce next year and that will be beginning its initial funding and fiscal year 2018.
I want to make a brief comment. Lucy, your question was terrific. Think about how these different elements of what we are talking about in terms of bringing a shared focus to the learning health system so thank you very much for that. That is the exactly the kinds of integration work we are looking for around this concept.
Sherry?
Not my usual question. In fact I had to ask Jen if it was a reasonable question to make sure. What about getting some of this into the medical school curriculum? For example, yesterday I heard that CBT is now being used as first line treatment for insomnia and that 90% of the folks writing those scripts are primary care and not psychiatrist. So employers are now implementing programs to help their employees on the well-being basis improve their sleep. So could we get something like what ever evidence comes out into the medical school curriculum to help folks to use them sooner?
I think that is an excellent idea. The notion of finding ways to get the professional education programs is something to think about.
That's a great presentation. Thank you, Francis. Obviously, you know that I'm interested in the quick K-12 mechanisms because we are part of that program. I just want to make sure I heard you quickly that the embedded learning health system investigators are going to be K-12 and not T32 mechanism and the current weight of their configured to create new knowledge will continue in the re-compete? Secondly, whether the K-12 that was funded for effective research will terminate?
I can't go into the details of the re-compete because it's on the street. I can say that as we think about our core programs and the opportunity to embed training and learning health systems and don't believe that any of the mechanisms of the table for thinking about that. Think about the balance within our institutional programs between producing researchers to generate new knowledge as well as other opportunities to leverage of those programs. The K-12 program that you referred to will and once those grants go to their cycle but of course those grantees will be eligible for future programs.
Thank you very much.
Jose?
I just wanted to indicate related to this a little bit that we have as a part of this agenda also engaged in discussions with the folks at the core rate to think about whether there are opportunities to align this training objective with interest that they have also for supporting this kind of expertise and development particularly as a think about it in connection with things like their program in which they are fostering evidence generation within different learning health systems and the value that doing training in that context may be helpful to them. I just wanted to make you aware of that.
I have a couple of questions about your presentation. One comment on what Paul said in the valuation. A long time ago the guy who ran the clinicals program at the University of Pennsylvania was a wonderful man name Sam Martin. He used to say if you want to make a silk purse, -- that's what you are doing. [ Laughter ] my two questions. First, who exactly is the target population for this? Is it sort of academic types are training or actually people were working in health systems already?
Both.
That is a great question. I think there is cross-fertilization to be done there. We see this in terms of the level of the trainees to be doctor and postdoctoral but Masters prepared folks within systems.
Okay. My second question is a follow-up on the when I made earlier. In your list of competencies, it might be sort of hidden in implementation sites, but I don't think so. This idea of sort of behavior change in social science of behavior change that's not there.
So what you are seeing their -- I apologize. It is some domains for subsequent competencies. They are going through a couple of rounds of a process to then identify those competencies such as that. I'm happy to share that with that draft set of actual competencies. You are right. That behavioral change component is important.
I know that behavior change was brought up and it is under that category.
Thank you, Lucy. I want to say I'm glad that the color development evaluation is well received in terms of that outreach.
I was on that committee to one of the cornet grantees and some of the concerns that the express was that their young investigators and all these affiliate programs did not have the necessary patient engagement and patient orientation centered skills. I was curious if you've identified with those are and how those fit into the competencies here.
So that's something that I'm looking forward to getting help with. They recognize that knees as a component of the competencies in systems. I think on Andy's point collaborating with PCOR and also the data systems within systems like PCOR sponsors would represent a wonderful opportunity.
Just a couple of thoughts. More comments than questions . This is tremendously exciting. As we look towards the future state of how health systems would be learning systems and how practices would be redesigned so that it would necessarily include automatic learning so improvement. It actually leads me to just iPods on what kind of opportunities this represents and some of the distinguishing challenges that might be in front of us. So the opportunities would be that one could focus on knowing how to manage populations effectively thinking forward to population-based payments as an example. It could also permit focus on those complex and multiple chronic condition populations, where how do we know if there's still a gap in care? We can measure with the system does and we can measure what the experience by Pete, but how do they actually match-up? I do also think that when we look forward into that learning health system in the clinical practice around it can really start to look and feel like quality improvement. So just to ponder how would we make the distinction if they are in fact would be one recognizing it is a fuzzy line already. The reason to make that point is CMS and others are really providing a lot of technical assistance and model tests to help practice redesign. So that's something to ponder. I think the one point that I want to make is that when we think about criteria for success of those who would be rewarded with these awards, it seems less satisfying to do a number count of publications. I think the impact if there's a different way to measure that, I think it would be really groundbreaking. Thank you for the work.
Thank you. I would be pleased to come back and talk about how we are planning to a value weight the PCOR investment. I think you are right on point there and the publication is dead because it is easy but not as -- making a difference in the programs.
Sandy?
First of all, I really think that the focus on learning health systems is terrific. One challenges is with a limited budget there is so much to do. I really like what you are try to do award you are doing in terms of integrating these different aspects of the agency but around a common well-defined problem or opportunity and especially when that hasn't been identified in a lot of the foundational work has been done. About the need for learning health systems and the Porton's that it's been adopted. It's really -- everything is leverage but there's a real need. I really like Francis and the training approach that you are proposing here because it addresses the question I had before and I probably should've waited. The ideal of training the people who can have a foot and understand systems operations and at the same time be able to understand the data needs to make sure we have evidence and we don't have data. One thought I had and I don't know if this is allowed to do this or not and I probably will be shot by my colleagues in the academic medical centers for saying this. I think that this might be the type of program that you may want to allocate or at least -- and who would certainly traditional ways by judicial review committee may not be as high but maybe doing really good work and have terrific -- on thinking here like regional health centers that have affiliations but aren't the main campus. On talking about a lot of the delivery systems that we have out there some of which are represented around the state. I really think starting a new program it's an opportunity to make sure that we have a diversity of models out there tied in with some sort of rapid cycle evaluation to figure out what's working and what isn't working and things. There are other things that could be partnerships. It may not always be the appropriate people to be the leaders of the partnership.
That's a very helpful comment, Sandy. I will say that we do have the flexibility to set eligibility criteria in such a way that we get at. So for example we said that eligibility criteria and research institutions cannot apply. We had a companion announcement, which was K-12. So striking that balance is where we are headed.
Naomi?
Thank you . This is in response to what share reaches mention and what you also mentioned with respect to the impact of health systems learning networks. It is more than just publications. At the VA we have 18 centers of excellence and one thing that we been looking at is how do we measure the impact of those centers of excellence in terms of changing delivery systems? These are VA centers that are academically affiliated with the major research institution. In the past, they have looked at a number of publications and numbers of grants and numbers of all these awards from the VA as well as external awards. Last year -- I was in part of that workgroup, but I have heard that the coin directors as well as the internal staff took a look at all of the other variables that they could use as measures of impact for research. Publications was one of them. Some of the other things were things like partnerships with operations, clinical services, policy changes in terms of clinical practices. So there are a lot of impact metrics that they came up with an we are more than happy to share those with you.
Thank you.
I just wanted to add a couple of comments and then I know we have one public comment. I think this is very exciting and I think there's a lot to be learnt about how to do this kind of work more effectively within and healthcare systems. This is some of the work that I've been trying to do in my own center. I thought I would reflect on some of the things that I've learned from that. I think what it has race for me is what we like to have happening inside health systems in the research space is -- goes against the grain of how we raise our young. So one thing is frankly the willingness to answer somebody else's question. So we select four and reward people to come up with and get funding for and pursue questions of interest to them. When you work inside a system you have to be willing to set that aside and answer the question that the decision-maker needs to have answered. It is striking how hard that is not only to get done immediately but to actually keep people from veering off into what they really wanted to do even though they have agreed because somebody is flashing money in front of them. The second thing that we've learned is it's actually helpful in terms of the partnership to have people saying what the decision is that they need to make for which evidence could be an important consideration and sadly not the only consideration, but one of the important considerations in that decision. That helps reframe things in a way that is more accessible to a lot of decision-makers then what's your research question, which is not how they think. The question is is it worth it for us to add this medication to our formulary? We can translate that into a seat ER question but what they need to know is about their decision. And actually changes the way you go about the analysis of the variables to collect and how you think about even doing the work, which is sort of two other things. One is they are making that decision on the timeframe. The committee meets on such and such a date and I need to know within this timeframe. You've lost the window. They will make a decision anyway. So then teaching people to be able to deliver on time, which is more of a sort of contract than a grant mentality -- is a very different skill. The skill is learning how to do the trade-off between the pursuit of perfection, which is a lot of what is true of research. Where can I relax and kind of not worry about things? I've got some great examples from some work that we've done where I know in a kind of classic research setting those five patients out of some million where we don't have gender identified we run those to the ground. We do not have time. We just let it go. It is these kinds of trade-off decisions and the other thing is that doing the work interactively. So the way that our center works is you have to have a decision champion for the work. That person has to be willing to show up and answer questions all along the way. Including are we still on the right track? Are we heading in the right direction? This is what looks like the answer that is emerging and if that is the answer how are you going to hold that in your decision? Is are some clarifying thing as kind of we are getting clear on this that will facilitate implementation. I have to say as a researcher who's been at this along time it makes me nervous to show folks the kind of rough-and-tumble of here's what the data looks like and here's what's not making sense. We don't think this is right but help us figure out why it is wrong. That kind of engagement is absolutely critical for having a stakeholder really stay as a stakeholder. On this thing that we've been talking about in terms of incentives, the way that -- for a lot of researchers even though in our health system they still face the same incentives that academic researchers face, which is about publications and grants. To the extent that awards are contingent on that behavior then that is a behavior that people are going to pursue. What we really need giving publication, timelines is for people to share their findings early before it is published and to get credit for activities that take a ton of time that isn't recognized, which is did you get into the guideline? Into getting into the formula? Things that sounds like Naomi and others have collected. I think finding ways to really explicitly reward that sort of conductivity and I think also the negatives. The other thing we forget research is we know always come up with definitive answers. We may do something that says not really clear that this approach is any better than what you are already doing. Somehow the not implementing or using that to decide not to do something I think we have to find a way to reward that because that often doesn't get published. So it may be almost more important to keep people from doing things that are effective as it is to get people to embrace things that are effective. I think -- I'm really excited at what you are doing here and I think you have a lot of chances to really think about the incentive systems that lead to this disconnect between research and practice and to actually change that in a way that will benefit -- not only AHRQ but the healthcare system. I think it even rethink and IH. -- NIH.
I agree with you. To me the publication pieces still important. Maybe we should think about with something like this different ways of effectively disseminating information. With emphasis on something that scientifically credible and evidence-based, but what I take from what you're saying is all the other things that we generate we need to look at. Maybe we need to also -- I think dissemination's fundamental to what government agency like AHRQ does an we need to think about what are the most effective forms of doing it and realize that is not just one pathway all the time.
What I would say is I am not [ Indiscernible - multiple speakers ] I am not anti-publication but in the learning help system here's the argument I've made. It goes against -- [ Indiscernible - multiple speakers ] exactly. Here's the thing about publication. People fear the prior release. The reality is that at least within the context of a health system, it is quite possible and this is where argue with the work that our center does. You can inform the health system of what you found and they can start working on implementing those findings while a piece of work is going through publication. The publication I've been arguing with our leaders is about our commitment to sharing with the rest of the world what we've learned. It should not -- we should not constrained ourselves to learn about it at the point of publication. So I think it is really finding the right balance. I still would like to see the work that we do published in the top possible journal and I don't think that's inconsistent with giving a sort of advance look. At least within our own system at what we've learned. Ideally, you publish something and we stay on that same day and here's a program we've implemented and here's what we've done. I think that sets up a different kind of dynamic. I absolutely think -- I got asked to actually within my own center give up the right to publication. This was sort of the politics of where work. I refuse to do that because I do think it's important. I think we sometimes get ourselves stuck on what that means.
I think we have to remember that there are other models of publication. We are unique within science that we have this finger roll. I think it should go where he is right now. [ Laughter ] it was done for good reason and there were good reasons behind it because people were hyping things before hand. In the natural sciences, everybody publishes their work and working papers that are almost identical so much of the social science you going to economics and they're always working papers. There often expanded versions or earlier drafts of what ends up being published and that information is shared more rapidly. I think what we are seeing in medicine is really a commercial aspects of the journal looking to protect their own financial self interest. That's another topic for lunch. I don't think we collect outside of AHRQ. We should think about new models for the medical publishing group.
May be the bigger point is that the field and AHRQ have to be able to embrace different models of what people want to do. I can imagine a very successful and very important and very impactful person within the health system who actually hardly ever publishes. I think -- is not going to happen in your shop a could easily have been -- happen in other places.
I have a person like that who said to me I don't care if we get credit for the work as long as the work gets taken up by the system. That's a whole different quality.
25 or 30 years ago it was a very academic field. It's involved in a completely different and very important direction. I don't think you want to throw out the baby with the bathwater. I think with the goals were of those people that got K words and it matters in whether the award was successful. In the future, you will be evaluating these people will be learning healthcare researchers. The people in 1990 if they want to be academics, you have to evaluating them on whether they were successful academics. Back --
We have one public comment from Lindsay from Academy health. Can you please walk to a microphone.
I will be brief. Thank you and members of the Council for your time and for your leadership today. My name is Lindsay I'm with Academy health. So I just wanted to take a quick moment to speak with you all this morning. I've recognize many faces in the room but for those were not familiar with as we are the professional society for help services and policy research. We value the contributions to the field. So Academy health has the privilege of serving as it is additional home for the friends of AHRQ, which is a voluntary correlation of 200 organizations that advocate on AHRQ behalf. So today during this public comment, we wanted to take up quick moment to update you all. You may be familiar with the friends of AHRQ . We initiated last year the #save the campaign. That campaign gained unprecedented traction from members of the community. About new attention to the agency as a result the people who were stumbling upon the #on social media gained about subway 3 million impressions and we have people not only tweeting but people writing editorials and articles and blog post. So I think you all should know that you really do have real friends out there who are really interested in AHRQ and the success. Looking at the current environment we do believe that in part because of the campaign last year that the agency was funded this year and the fiscal year 2017 bill in the House. Despite this lower level this was a really important step in we do think it laid a great foundation for moving forward. With respect to the current appropriation cycle, we had two TR colleague letters that were circulated. Those were from representative Chris from Maryland and then center from Connecticut. We also worked really closely with editorial from a representative, which was published both online and in the current versions in early October. That was fantastic. We are continuing to build champions on the hill and there's a new excitement and energy about the agency that we have not seemed purposely. I do think that is a reflection of the leadership. Very exciting things. We look forward to seeing how the developments continue and then I will just and by noting if you are not involved with the friends of AHRQ , we hope you when your organizations will be. I will end by just emphasizing our support and noting that if the friends of AHRQ can ever be a resource for you all, I hope you will call upon us.
Thank you, Lindsay, and thank you for all the work that you do.
We will take all the friends we can get, right? Here is a game plan for those of you whose name Andy, José, Paul, Mary, and Sherry you are going from here to have your memorial photo taken and then you can go get lunch. You are having yours taken because you were new and you were not here for the new member photo. It is all very confusing. We have your wanted poster, yes. [ Laughter ] it was his comments about economics. And then everybody else can go get lunch in the cafeteria and we will attempt to reconvene at 1:00.
[The event is on a lunch recess. The session will reconvene at 1:00 Eastern Standard Time. Captioner on stand by.] [Captioners transitioning][Captioner remains on stand by] .
.
[The event is on lunch break to reconvene at 1:00 EST] .
[Captioner remains on stand by] .
Did you want to start us off with anything?
First of all, thank you, so much, everyone for being so engaged this morning and thank you to the members of our senior leadership team for their presentations, and what I was hoping we would be able to do with the time coming up now is to, both continue to get to the degree that you have things you want to express about your support or questions about the concept of our focusing on the learning health system. I already started to get a strong sense that many of you are endorsing that notion, and what would be particularly helpful for us is too really get your input, thoughts around strategy to execute on this. I think there are a lot of things to take on so getting advice about, for example, we have had thoughts about if we were to co- create with Health Systems around this, do we think about creating some kind of learning network with Health Systems? If we do, who are the right people in the Health Systems to engage with? Our their activities underway in the space already that we should think about piggybacking onto versus starting are own? That kind of strategy would be very helpful. I also think that getting your advice about maybe the order in which to think about. A lot of things in this space. We have already nudged forward and our thinking about making training one of our early significant investments in this regard, but also with regard to things like the tools in ways we are thinking about engaging with not only generating the evidence but applying it back. I thought there was a tremendous amount of expertise around this table, people who are working in ways related to this that we would just benefit tremendously from your advice and guidance about how to really move this beyond, that sounds like a good organizing principle, in two steps we might really build toward in terms of a work plan that we can do together, and to make sure that we are thinking about the right partners for some of this work as well. I guess those are some of my what I hope to accomplish during this time. Our goal was to be available here as resources to answer questions, but, it's less about us presenting and more about trying to foster your thoughts about how to take on the challenge and make sure that we are bringing a lot of added value to it.
Thanks, Andy. I know Lucy wanted to take us off and, David Ballard, I also have you on my list next.
My comment was a carryover from the prior conversation about publication and the extent to which embedded researchers are engaged in publications. The only point I would make is we tend to silo when we talk about these things. Thus we are and delivery systems and have the struggle of, can you get the results out to the decision-makers in a timely way versus waiting for publication. But researchers in delivery systems need to publish as well for purposes of scientific stature because many of those investigators, in addition to doing unsponsored work and the delivery systems are also speaking at compete for external funding. I think those needs exist in both places. We are also required to disseminate in many other ways though.
Lucy, you remind me of one of my other wish list things which is to find a better way for what we call embedded researchers to get credit for internally funded work, which often, the other reason some of our best folks do not embrace that is because they do not get the same kind of credit when they go for externally funded research. The internal funding is seen as, of sufficient stature to get the same level of credit, so they both don't get credit internally. They don't get credit in promotion decisions internally and the work that supported that way. In some ways it's our system's highest compliment. We think it's worth putting what we consider to be scarce resources to get them to you so that you can make a difference. Just thinking about how those things might get a somewhat different treatment. With folks making external funding decisions.
I would like to talk with you offline about that.
David?
Thank you. I think this learning help system focus can be transformational for AHRQ. When you are looking at declining federal funding, if you could create valuable partnerships with health care systems such as mine, that could bring substantially greater resources than you have ever had to your agenda. Just from a personal perspective, to make a comment about three things sort of funding, data, maybe operational mentoring. I felt the various training programs you discussed, I think having health care system partners bring to the table to the people you are funding through these mechanisms, additional financial resources to help support their work, access to data, and also access too, not so much scholarly mentors but operational mentors. I will be very brief. My [Indiscernible] dissertation was funded by the national Center for health services research. It fast-track my work at Mayo Clinic because the dissertation was based on using a Mayo Clinic data system. It allowed me at a very early stage in my time there to become very familiar with the Mayo information systems that would help us support such work that I did. Also a very important part of that was I was mentored by the Chief Medical Officer @-at-sign-at Mayo, Dennis Cortese who went on to be CEO. Dennis is support for that work was really critical for my success over my next five years at the Mayo. As you think about the health care system partners, they can bring financial resources to the table. B, having access to data systems early for these doctoral candidates and early career people is important. Thirdly, having a plan for operational mentors so that there work gets connected to the operations. Lucy and I, for example, in Salt Lake City have a real struggle with the high value healthcare collaborative as how this works gets translated into our operational initiatives.
I'm not exactly sure if this answers your question but to let you know, we've been working -- we created a Committee called, the Executive familiar -- the Executive Committee of value purchasing in a year plus ago. That evolved integrated systems and [Indiscernible] health plan. [Indiscernible] CIGNA, [Indiscernible] and Kaiser. We are just now releasing -- it was on how to move and accelerate the adoption of value purchasing amongst employers work one of the biggest issues around going into ACOs is what are they going to get for the money? What should they get? We have created a journey map of beginning, growing and mature ACOs. What you might expect along the continuum of things like governance, network development, care model, consumer experience, technology and finance. The priorities that this organization, the Committee came up with by consensus. If that's helpful, I am happy to share the materials we have created. It's going to -- the journey map is in the public domain as well, and to let you know what the priorities around that. It includes things like guidelines, shared decision-making. If that's helpful, anything that will help accelerate the move -- employers are already paying the care coordination fees but they are not jumping into the models with enthusiasm. So if there is things you can show are embedded that will help, we are more than happy to share.
I think that's great. Thank you, so much, for that. I think its something we may want to talk more about. Not necessarily go around but the idea of developing indicators, or ways of knowing about sort of progress toward some goals [Indiscernible] Health Systems. It sounds like something you were working on is something we would be interested in hearing feedback as to whether this could be an productive area for our work.
[Indiscernible] open dialogue between employer and open contract [Indiscernible] or through your health plan to say, I have a population in Seattle, that's an example. Which ACOs are available? Where are they along the continuum? They can do a consumer report and [Indiscernible] saying we are here and here and here along your journey map. That's just to open the dialogue.
I think we highly recommend group health.
Jose?
Andy, did you ask about potential models for training these people? Is that what you were asking or were you asking something different?
Frankly, we're open to feedback around any of the issues here. I was taking a slightly higher level step as saying, were talking about the learning health system and the focus of agenda and trying to figure out -- we have, I will put on the table, training is an area that we see as potentially an early investment in the activity but really want broader feedback about the correct ordering of how to go about this broader agenda of learning Health Systems. Do you agree with our thoughts about A, building training and the direction but other concrete steps you would advise us to take in a sequential way to try to execute on the idea of a learning health system based on what you see is priorities or opportunities?
I was going to comment on training issues. Maybe I can hold off until later.
Mary?
Can you hear me?
I have been listening to the discussion all day, all morning, I should say. Very interesting about learning health care systems, but in terms of the terminology used with the system, it seems to me different interpretations to different people. I don't think it should be narrowly focused on integrated Health Systems. As you show in the previous presentation, only 34% of physicians belong to Health Systems. There are large numbers of individual practices and so forth. Another way you can look at it aside from [Indiscernible] physician practice is through communities. CMS has charged the QIOs to develop communities, collaboratives working together across the continuum of care to reduce readmissions in care coordination. CMS is testing Accountable Health Communities. You can think of it as broad as a community. You can think of it as small as individual practice. I think all of these levels need to be a continuous learning organization, and we, as a QIO, of course, think of that continuous quality improvement. I think you had to start with individual practices and teaching them how to do continuous quality improvement on their own, collecting data and answering their own questions and so forth, in order to improve their care. I just want to point that out.
Jim?
-- Jan?
Ditto to what Mary said. Every time I put my card up somebody says exactly what I was going to say. Again, I want to challenge us to think broadly because there is a lot more that impacts our health than healthcare. Ring in the community and the social determinants and, absolutely the primary care aspect into account not just thinking of hospitals. The hospital is an easy place to embed a researcher. A community that stands across multiple spaces more challenging but I think we should challenge ourselves to do that.
[Indiscernible]?
Andy, in terms of messaging, branding and re- energized kind of focus for AHRQ, have you thought about how learning health system fits with the core kind of message? Some of the figures that are being presented, there was a place for biomedical research. You talked about PCOR patients outcome measures at the end but in your diagram, I did not quite see where it fit them. If somebody were presented with learning health systems and patient-centered this, how would those integrate into your messaging?
I think this is a really important aspect of things. I will tell you some of my ways and I am conceptualizing and would welcome thoughts from Panelists or others here. I think one of the challenges that entities have and I also want to address, Mary, you are saying we are starting with integrated systems. I think we are looking at vertically connected organizations as a starting place for the workday described, but I don't know I would insert the word, integrated. In fact, I think one of the things we are hoping to assess is, how integrated are they? What is integration mean and so forth? That's one aspect. To get your point, I think the goal of learning about best practices from evidence, I don't think the end result is everyone with the same conditions; therefore, everyone gets the same treatment. What you want to have as a systematic way an individual with similar conditions are approached with regard to what the evidence says, and our given information and an approach about the information is shared and discussed with them so they can make personalized choices that makes sense for them, given other contextual durables and so forth. I think what we are interested in is understanding capacities of healthcare systems to be able to provide that, not only be aware of the evidence, and in some cases, generate themselves, but also adopting from outside, but applying in a way that has a systematic consistency. Not in terms of being reflected as, everyone getting the same treatment but the choice individuals are given. When it comes to things like PCOR, comparative effectiveness, I think that's critical information that, presumably, patient and the Health Systems would want to know what is the evidence about the comparative effectiveness of different kinds of treatment options so that, that can be Incorporated into decisions that are made? That seems to be an important piece of the evidence. It's not in all of the evidence we have but certainly seems like a critical part of it.
I wondered if we could come back to a bit of a discussion about what the opportunities might look like with PCOR, in particular. For those who don't know, Jen and I are involved as well as Lucy with the PCOR national patient-centered outcomes that were, like that, research network, right. The name of it and it's handled PCOR net. I have had my press people how this goes to gather. Anyway, what's interesting, I think, about it is there are something like -- so there are both clinical data research that works which come out of Health Systems very broadly defined, and really kind of, I would say, including, maybe not so much solo, but definitely smaller practice, federal qualified health centers, integrated Health Systems, academic medical centers. I think it's a good variety of different kinds of [Indiscernible] systems or care delivery VA is a part of one of them. And then there are 13 of those networks that include more than 100 institutions, and then there are the patients that -- patient [Indiscernible] research 20 of them are barely focused on single diseases, although some are broader. Together, they actually have worked a lot on how we identify questions that are important to patients, and not that I would say we've got that figured out perfectly, but certainly a lot of work as been done. And then the systems are developing and our kind of creating data in line with a Comdata model. I think there are something like -- I should know the number off of the top of my head, to 19. 40 million lives, somewhere between 40 and 70. It depends on how you count it and which they you are counting it on, anyway, lots of lies. Both as Dave was talking about in terms of having a broad set of institutions to test even definitions on, we, in the group that has worked on the Health Systems demonstration area have recognize -- recognized and need too, at a minimum, characterized Health Systems so that when you're looking at results or you are thinking about implementation, you may have some kind of a framing device for how to figure out what kinds of systems producible kinds of results or some other kind of characteristics. We also went through a process with our Health Systems leaders to identify questions that were critically -- critically important to them. To Arlene's earlier point, we have some work they came together that we try to summarize into at least the broad theme of things we heard across our Health Systems, common ground, and some sense of what they were interested in. What they were interested in learning about from other Health Systems.
[Captioners transitioning] We also went through a process to identify questions that were critically important to them. We have some work that came together that we tried to summarize into some broad themes of things we heard across the health system. It was common ground. Some sense of what they were interested in learning about from other health systems. I do think that in the learning health systems area there is this -- what do people see as value and compare and contrast and collaborate and what do people think they just have to do it home. I don't think everything fits into one bucket or the other. I do -- I am wondering what the boundaries are and what the opportunities are. It seems like this is a big thing that has had a lot of money invested into it and it is now slowly opened its front doors. I think we have some windows open right now and have plans for a more formal opening soon. It seems like a great opportunity to collaborate.
I think it is a great suggestion and would love advice around that. It does seem like that -- Dave I would ask you to chime in as well. One of the challenges we feel is that we will make assessments from often secondary data sources. As others have raised, there is the Rex that we could get an overly simplistic feel of how health systems actually function as health subs best systems. Being able to develop relationships and I think PCOR net would be able to develop more of a cocreated way of pursuing this work were there would be more engaged partners with those who could give us more insights about those systems and test out -- here is what we would have said about your health system based on the secondary data work are we even close. Those other things that we could have gathered about you. How do we fill in this picture better. Working with perhaps certain health systems that are already engaged in the process might be a really helpful way to start to advance our work. I don't know if you have further comments are not.
Only that one of the reasons that we decided on these larger vertical systems as our starting place is in part because of the feasibility of being able to even identify and link individual providers with the system and then getting enough secondary Dan -- data to characterize them. We recognize the inner workings and some of the complexity that was described earlier about forming relationships and verbal types of relationships as a product line. They are difficult to characterize but they describe what is going on. We need this interactive process to get there. From a timing standpoint we do have a couple primary collection efforts that will be launched pretty soon. The timing of the content of those surveys and making sure that we don't step on toes in the field and that sort of thing, certainly thinking through how we could start to triangulate in from what we are learning, some of it could be with a one-off survey that could give us a lot of information on where to go next. It is a tremendous opportunity.
I would just add that when you're talking about systems the most common answer I got when I first got to Kaiser Permanente and and asking how things work, it depended on the region. Then it depends on the hospital or office building. I think it is important as you do this work took not miss the fact that most of this happens on the front lines. The workflows can vary. I remember asking early on is it okay to admit that we have variation? The external view is that everybody does everything the same way. In case you haven't heard, that isn't the case. Understanding the data that we see inside the electronic health record is often a matter of understanding pretty local workflows that either produce the data or tell you in a forensic accounting way which of the 10,000 clarity tables you might actually find that information in. That is the way the systems are set up. I think beginning with integrated systems is helpful. You haven't easier -- path to finding information. Anything that looks too simplistic, the VA would say the same thing, we all do the same thing, that would preclude that you're doing it wrong.
That is an awareness that we have. We realize that as a starting place. The COD has a broader definition of systems. There is a conundrum that the limitation on our understanding of what is going on is precisely that we have had to look from the outside and squeeze as much out of it as we can. When you really look at what is transforming inside the systems, having worked in them, you know you're not explaining what is really going on. We really have to go there. I totally agree.
You had a comment?
Yes. I hope this doesn't take us away from this important conversation. You were talking about starting points. By think it is going to be important to frame what you are hoping to achieve through the learning health system and I always think it is great to have a little bit of a focused/focus. Your goal was to do something you think is a foundation to create the kind of care systems that are addressed in these to meet the needs the people best meet the needs of the people. I am wondering if having a target population as you set out on this journey that might be served by multiple vertically integrated health systems that might be served by primary care practices. I know you're talking about starting within the large vertical systems but maybe the co-creation model is that you engage on the outset all of the other structures that clearly need full investment in terms of building learning health systems. As you are thinking about how it is that you will move a learning health system and maximize on its contribution, you're doing it with smaller systems that may not be vertically integrated who are part of the co-creation so they can see how this could move in and they could contribute to the actual design of the work because they share in common, complex chronically ill people.
I guess I would say the co-creation part of it is making sure that all of the ultimate stakeholders that you want to have engaged are part of the design team. Even if you want to launch it in a structure that your ultimate goal is to spread it and maybe as you do the building prototype for primary care's etc.
I guess I will pile on on what you have been talking about. I think on the use of secondary data for this research in our opinion they are limited because typically what you get our structural features. Those will only take you so far. They might not take you very far at all. Dave has presented the information on the three centers of excellence. UCLA is part of that. When we wrote that proposal, we very specifically wrote it from the perspective that secondary data would only take us so far. In fact we wanted to focus on trying to do the best we could to go inside the organizations and try to measure all those things that normally are not measured but management -- matter a lot. That is the organizing principle of the center. We hope that will be successful. I think to the extent that I am sure everyone believes that more or less -- but what is really going on inside the systems is what matters both in how they perform and how they change and whether they become learning systems etc. and how they change and whether they become learning systems etc. Then I think that the implication there is that there is no choice as you think about your program for developing the scientists are going to be embedded in learning health systems work there is no choice but for someone to develop partnerships with those systems. I think a big part of that training will have to happen inside those systems. How that happens and who develops those partnerships got you probably have a much better idea. Certainly academic institutions could do that. I think that is going to be a crucial element Partnerships.
I skipped over you, I am sorry I wanted to comment on what José said. I thought the same thing. There is performance across health systems within. There is also that within the systems. The same way that there was the lack of evidence as I have told you before, you often have in the knowledge of what is the best evidence for intervention for different minority populations. Just falling on [ Inaudible ] and targeting populations, maybe there's a way to really use this data to get some key priority questions that we don't have the evidence of them diverse populations so we can [ Inaudible ]. Just reacting to what José just said, I think we need to think about what kind of research we need also. What I'm hearing from you is excessive research. We need to think about how we will study this. Maybe that is something that has been done before, some of the methods development. Also, somebody has worked with this data for a long time and the limitations of the data and the frustrations of it is that thinking about how with EHR is a new technology we can start capturing more unstructured data in using that for processing. There's a lot of potential for advancing methods to be able to study this at different levels.
I think are comment is really am -- interesting and him Porton. We need to bring in the disparities. More and more I hear conversations about high-quality health systems becoming involved in bash and the social condition -- in the social conditions in which their patients live. I guess an interesting dimension of this may be to what extent the high-performing health systems are. Not only those that bring clinical knowledge and apply it to the right patients and taking their preferences into account and all those things but also the things that engage outside their walls to try to address caught in an ideal case modify the conditions under which there this advantage patients live. I don't know the answer to that. That throws a new dimension into all of this and how we can centralize the high-performing health system. I just finished my service because it is over. That was funded to develop a report on this issue of adjusting performance measures for socio-economic risk factors. This idea was kind of always in the room. I found that interesting. In any event, the comment is maybe that is an additional dimension to think about as you move forward, the scientist who are going to this. Do you need some people who will understand social determinants. There are three quarters of the effect on health.
Think that was a nice segue to what I was going to comment on again. I was struck by -- I was thinking about vertical health systems as a place to begin. I really want to challenge us to think beyond that as a health system. Maybe it is partnering with employers and payers and health systems. Many patients who perhaps our most vulnerable are invisible because they are not able to access them really thinking about the data that we have in this -- and the secondary data sources and the data we don't have and how to make that whole. Some of the ways that we are doing it are integrating health insurance claims data. We have some studies where we have 10 years of Medicaid data. That is linked up with hospital data and it begins to bring in the geographic data at the neighborhood level. Ideally then we are collecting that information at the patient level. We're thinking in a multidimensional way. I want to caution us against putting people in traditional hospitals and saying they are learning about health care. Maybe they are learning of the dysfunction of the service. What kind of already know that. No matter how efficient the hospital is at bringing people in and not read knitting, there is another aspect of their house -- health that needs to be incorporated in a different way.
They'll be?
-- I agree with everybody. It is the importance of the determinants of health. I remember before I came to the VA, we were working on a model. I believe it has been released in terms of looking at the social determinants of health and saying whether at a community level, we can address those factors rather than just the medical components. I totally agree. In reference to the question you had got what kinds of things might week come up with in order to develop the health learning system model? At the VA, I have been there for various short time. I was there seven years prior to my getting back to the VA. A couple things have changed. That has actually led us to new funding opportunities. I know that it has been mentioned, a whole slew of things that you are about to release. There are couple things that made us change and shift our funding priorities. Instead of a condition focus, not diabetes or cardiovascular disease, what we have seen -- we're still doing those areas of free should bash research but we have now shifted to organizational changes. Another area is in terms of more partnerships being developed. Funds that develop their partners -- that we partner with other researchers. Previously there was no relationship. They look that develop thing information for certain diseases. Particularly operations in clinical side. One of the things that we funded most recently was the partnered evaluation. It was research implementation. That has required -- you have to partnered with services. They have to at the very beginning work with the partners copy clinical side to develop the research and design and evaluation. That has been very helpful in terms of implementation because of something is found to be effective, you have the clinical partners already there. They could then start to implement and scale up if something is found to be effective. The other movement, it is not a movement away, maybe less emphasis is the secondary data analysis. We found that we have a lot -- the VA has lots of data. We have tons of data, the data. It's just data. That was the first step towards developing new research priorities. We had to take what was being collected. We're finding that the new groups of researchers are using more clinical trials, mixed designs. They are actually going out and not only looking at and collecting outcomes and looking at secondary data but they are actually going out and interviewing patients that are in this case and providers and medical executives Business partners etc. Those are the three areas where we have made significant changes. The funding that we can't grant -- we're not in the grant business. We can only fund awards by statute. In the four months I have been there I have clearly seen some changes in the kinds of funds we release. The third point and then I will stop talking, implementation science. We are funding a greater number of awards in that area.
Can you just say, to develop the science or towards that require recognizing that there is a science to implementation?
It is the latter. There is an implementation science procedure already and how did they implement it and apply it to certain areas of need for the population.
Alice?
I just want to make a comment from the patient's point of view. In 2015 -- we are on condition. We brought together the science and stakeholders in one room. We had an end goal in mind. We were looking to create a patient engagement or research platform based on the patient's wants and needs. When we're talking about bringing all the stakeholders together and creating that map, maybe what you do is hold a day or a conference kind of where you bring -- if you have the funding available, bring in the stakeholders and then, we had three moderated sessions. It would depend on how you wanted to develop it. By the end of the day and a half we had developed questions and framework is started on the path forward. Those people were -- we were in subgroups and over a six-month period of time we were able to choose our design, platform and move forward. I don't know if that is an option to do something where you really have another meeting to bring stakeholders together.
I had a couple of thoughts. I think one of the important learnings from this would be what treatment or services are not being paid for with the current coding for fee-for-service and what are the best practices that should, in an advanced payment setting be suggested to be covered. Listening to the primary care medical model for oncology, Bears care management and talking and education and integration of mental health in some of these best practices that are not coded or not paid for. That is a stumbling block for making system changes. I want to make sure that we understand what is missing in the mix. I would also say that from a transparency perspective we had that meeting last year with the vendors out there, they are keepers of great data. Specifically they would cast light at a conference. They know based on age and ZIP Code who will have diabetes. Opioid misuse. They see that in data. They actually published something on that. Making sure that's gets funneled back to vehicles of transparency or quality, that way it can be added to those models as well.
Sherry?
I think what I hear reflected in multiple statements is this is a complex system -- complex issues. We know what we can measure and what data we have. There is an equal and more challenging part about what we don't measure. That is either because we don't have the data or the people who we might have interest in are not in the system so they are not contributing data. There is a lot to the missing this that I think it is important to just know that we know about this but we may not know about that. On the point of what we don't have reflected, I was thinking about this, the socioeconomic factors. We know dual status means a lot in so many ways. One of the things that we do have a direct way of measure consistently is function. How do you actually measure and include in the analysis something that would be as much as functional limitations. It is complex in the here and now. Also, to share his point, there are new services that are covered every day. Or, in the middle of a measurement. Or a new determination got national coverage determination that might come into play for new policy, right then. This is a long-winded way of saying that with such complexity, I think maybe the current analytical approaches are sufficient. Maybe they are not. I don't know. It is hard to measure something is critical as culture of safety. We measure what we can measure maybe there is something -- there might need to be new approaches. That is just food for thought.
I think you have surfaced some of the things that I was hearing and several of the comments. I want to say back a couple things that I am hearing it frame up what I think we need ongoing help with. I think several comments raise the fact that, if you end up biting off -- fighting off on definition there are all these other problems. Are you sure you're fighting off what is just under the lamppost as opposed to what is most important. I think the reality is, someone else said it, we need to develop brand around this. There is no proof of concept yet. I don't think out there of how to actually do this learning health system thing. If Arc is going to get traction, I guess I am just reflecting back. Maybe this is consistent with something you were saying. We need to have -- what is base camp going to look like? What is our first successful part of the ascent on this road? There is tremendous complexity in the system as you were saying. To some extent what feels perhaps, this is where I would love some input got targeting some organizations that already have a nidus of starting to do something from this bench on this from the inside and us figuring out how we can help them might allow us to both find ways that would bring added value, learned about, is their concept here where using knowledge and training in other kinds of strategies can actually allow organizations like the ones Beth represents that are seen as on the edge of doing this. There is already a lot of variation. We know all that. I guess it seems more challenging even though the right thing for soft affixes some of the challenges you are bringing up, I think we need to have a strategy. That is where I would love additional input on whether to build up a brand. Are there right start -- right targets to go for. I think marry you may have been saying this but you are also saying make sure that everybody is involved in the communications as a way to make sure that it isn't the way we start off with this first group and that is what we would end up with. That would not be satisfactory. I feel like we also need maybe a lower bar to grab on to it first to see if we can build up a brand and so forth. I am pushing that back out to you to respond. How would you strategize? Do you agree with some of what I am raising here and do you give us other advice around that? You know that we share the cause of wanting to address social disparity. These are all exactly where we want to go work we also want to make sure that in the process of trying to build such a complex response we don't think and lose the ability to ever get into this space. That to me -- maybe that is how operations come in contact with research. This is how comes out for us. That's where I need help.
I would say one of the very first requirements is demonstrate that you have a robust laboratory already built. I see health services researchers all over the country spending 90% of their time trying to build their laboratory. Basic scientists don't usually have to do that. They show up in they have OLAP. They bring millions of dollars worth of equipment. We spend our time building labs and building the exact same data sets together over and over again.
I may data sharing agreements have I had to repeat for one more study. It really is Tell us what your lab looks like and convince us that it is mature and robust. Again, that might be a hospital but it might be health services researchers that has 10 years worth of data. They have are ready these engaged stakeholders and they have the lab and it is mature. That will look very different in different settings. I think we can do both as opposed to either or. One of the most important things will be the mature lab in the beginning.
I would like to say that we do have examples. I am one of the founding members. I think you're founding number as well of the Fellowship program at HR Q. It was started in 2012. We have trained 26 fellows. There are 13 develop -- systems that participate. I think that is one place to begin. End of our, I know David is all McCall, the collaborative will now be a site for the delivery system Fellowship. It is a lab. We have that agreement across all the delivery systems and we have claims data. Week have examples. I don't think we are far behind.
Good.
I want to ask again whether it is necessary to keep the traditional primary care focus on this learning system. I reflect back over many projects. It was also quicker and easier to engage practices with ophthalmology, cataract care. We could get improvement cycles going in those kinds of prescribed areas much faster and great examples and proofs of concepts that could be then applied to more complex areas.
I actually have one response that is very useful information. It looks like you want to comment on that. One of the other things that has run through my mind around this work is, I think this was brought up to me with regard to some of the work that we have had great success with it safety areas, we often are able to do things in a narrowly defined area. Yet I think, we make impact. Is that the way that we have to continue to make impact which is in these incremental kind of on clinical problem at a time? Do we imagine that we will get to a point where the organization has a systematic approach to the clinical area that it knows how to do that? I would welcome input on those of you who have worked on the frontlines of these kinds of things. I bring up a lot of things because it seems like that would be a great goal if we could just have organizations that were already doing this and did not have to anticipate every clinical issue. Maybe it does require what you are talking about. I think that is a challenge that I am trying to sort out. Can we think about doing this organizationally versus cotton no doubt we have to take on the eye problems and then the hip problems and --
That was attention we faced all the time. Don can help us look back on the pursuing perfection collaborative initiative. On it -- what are the lessons from that?
He is not on the phone.
Pursuing perfection never got there.
I am hearing lots of different types of activities. I am wondering if it would be helpful to have a framework to tell a story. I think there may be low hanging fruit. You don't want to go off in the wrong direction though. I think the real goal is to switch from disease centered patient care. If we are headed in that direction, we don't want to get derailed. Also the need, the role of the piece of this that is data development, we need the right data to be able to measure the things that are important. The work we are doing with the trust fund is using physical functioning as a use case to be able to get back into the EHR. At the same time we are developing a process where we have standards and adapted to other domains. You could get other things down the road. I think if we do some parallel things about developing specific areas to improve our capacity to measure and develop a process to expedite improving the capacity.
Mary?
I wanted to respond to your earlier question about getting started again. I actually think the strategy you outlined and especially given what we have just heard in the most recent comments that there are examples out there makes a great deal of sense. You should begin to work where there are already -- where you can help identify the pillars of a learning health says that as health system and perhaps even partnering with communities that have really been working on that. The collaborative that you heard about. I do -- I would also like to comment, I think Arlene's notion that we have to move beyond a simple issue and art -- our capacity to really work We have to narrow the needs. That is something that I hear about in every high need, high-cost advisory group that I sit on. I think you really want to be taking on building the prototype for a learning health system focused on the very complex health problems. That is what everybody is trying to figure out. That is where I think you will have the greatest capacity. To the point of messaging, I think the optics will be very important. Many times we look to systems such as those represented around the table today as where we go for learning. Others don't feel included in this process. The opportunity here is actually in fact the co-creation of the language you use it the beginning to really make it clear that ultimately you are about trying to foster learning health systems across the message structures that exist to deliver healthcare not just to vertically integrated health systems. Even though I know about the variations and so on. I think that is going to be central. This is not -- this is a process that you want a lot of people engaged in beginning with stuff that is working to get those pillars identified so you can help move them too many people. People know that from the outset. People
/. -- From the outset.
This may be more circumscribed than what we are thinking about. One area could be research and implementation related to disparity reduction. I also think about this particularly in light of, if you look at CMS value page pavement initiatives, people like Mike colleagues would say those are leading to greater disparities. For a RC to work with healthcare systems around the country, I think that could be very important.
Mary?
I want to underscore and echo some of the things that I said earlier work the fact that messaging is important to say to the world that you are starting with a horizontal system but you are looking for information that can be applied to other types of entities as well. I also want to echo that, because of the complexities of the delivery system out there, we are no longer -- have the luxury of addressing one issue at a time, especially with the chronically-year-old Nashville -- chronically ill. That is where frontline providers are struggling. How do they apply a diabetes. That is a lot of the things that this frontline people are struggling with. The other thing is that with respect to the health systems around the table, I have high regard for these. When we are in these learning in action network in trying to share best practices with the physicians and the nursing homes and hospitals that we are working with, whenever we are asked -- asking a Kaiser person to talk about the best practice, the answer is always it does not apply to us. We cannot do that. We're trying to learn from this high-performing system that can be applied to other organizations
I hesitate to enter because I am a statistician and not practicing where you are. I was going over Dave's slides. I was thinking of the three centers of excellence and the coordinating center. That I looked at the last slide. That is the next step. It says create variables and public use files, conducting research projects disseminating data and findings. Ho-hum.
Frankly. If that is what is going to happen out of this, it is maybe more of the same. Maybe what -- maybe I don't understand what the centers are going to be doing but maybe there has to be centers for -- centers of excellence for practice that follow-up on the centers for action that you may be working with CMS and others to help fund to see where this can really go. Then, on the other hand, I was listening on the comment of looking at specialties. To me, I think that is the easy way. Maybe this is the time to really look at, not downstream but where the problem is and that is in the primary care physician who is seeing people like me with multiple conditions and having to figure out how to do business with them and how to convince them so that they don't have to go to the specialist at some point in time or at least delay it.
I think you can put your [ Inaudible ] on some of these acts taken -- activities. You will always be improving. These are things that are sanctioned by or at least by CMS and by other professional groups and then get funding for those activities to take place. Then look at the results of those applications so that people don't look back and say well that is UCLA. We will never be able to do that. That is -- how can my practice do that? Take it there. Give it a try.
I don't know what I am talking about on this but I had this feeling that we were getting a little bit about research for research site. My organization gets involved in that quite a bit. That is [ Captioners transitioing ]one of our problems.
I don't know if that is helpful from a system perspective, companies like village and the who is doing that and Texas and Indianapolis and now they are expanding into Chicago. [ Indiscernible ] health care which is in this healthcare, and again, just trying to create a structure for those primary care doctors that didn't have it before. Very fascinating transformation on the ground that we've been trying to figure out what they are doing. If that is helpful, they are new in that space, may be very interesting to study.
One of the things that strikes me -- let's see if this makes sense. In some ways, I actually like the framework because it probably is because we are working with a lot so it deals familiar to me. I think it might be helpful, and I think we started to do this here today. Think about what do we mean by that, and what are the contributions? I do Inc. that having a framework could be helpful, quite helpful, and actually, -- I use this term a lot which is a [ Indiscernible ] framework. Is really having a framework for knowing what we want to know. It is thinking ahead of time about what we would like to learn so that as people are doing work, and we have been talking a lot, and I know this is not a unique term at all about [ Indiscernible ] which is we do thousands of pilots a year, and we don't learn anything from them internally because you know all the reasons. Anyway, one of the things we're working on is to try to develop some frameworks for some of the key areas in which we are trying to learn how to do a better job at something. These are usually fairly [ Indiscernible ]. To try to agree on something so we can at the end of the day at out some of those small changes and make more sense out of them them we can today starting with finding out what is going on out there. That is investigation. I was thinking -- I'm sure this is an exhaustive list -- but we talk about content areas so our ability to learn about how to do fill in the blank better.
There is learning about the process of learning. So what do we really know mean? Could you give anybody some guidance on how one would go -- then I think learning about what -- there is the learning about your own practice from your own data. There is learning about things that are mode -- more generalizable. We get a lot of questions about what is the best systematic way so really has to do in which the way we organize our system of care no matter what. That itself can be [ Indiscernible ] but the process by which we learn -- another couple things are useful methods. Here is another thing that I think the R01 funding model has wreaked havoc on in America which is the emphasis on innovative methods. Frankly, I think if we just used old boring standard approaches and did them well and repeatedly, and we -- my team works a lot another group that works with me on reusable code and reusable data infrastructures. This is Jen's laboratory idea where it isn't to have some cutting-edge methodology. Is actually to take a standard methodology and to use it appropriately and often and with the same code so that you don't spend all of your time reinventing the wheel. Even having some guidance for people. If your question is such and such an appropriate method is such and such. It needs to have these attributes. Simple things like that.
Then there is this sort of data for learning. So what is it -- there, I think this is a little bit what somebody was getting at and now I've lost the trail on it. The things that were most often missing we're trying to learn thing about care delivery innovations is context. So we can sort of say we set out to improve functioning after joint replacement surgery. But all of the stuff that is the secret sauce to what we need to that, who was motivated? How were they engaged? How do we get the patients and the doctors on board with this? That stuff is always often lost in the ether and then when you go to replicate something, and we have lots of examples of this. It's sort of falls apart because you are missing the critical elements. It is really not about the engineering part of it. It is the it is not about the hardware -- it is the social engineering. We don't do a very good job of describing what it took to pull it off and how you could get that to happen someplace else. Or if you can't, I will use one simple example, we find over and over again even to have a successful clinical trial done takes a clinical champion that is local and who believes in the project and who will run his or her colleagues to ground to get them to enroll their patients or collect the data they need to collect or whatever. I have gone to the point with some of our projects where you cannot deliver Anchorage champion. That is someone who answers my emails within a week of me sending them. We are done here. We can't make progress. When people can only talk to us once every four or five months, it is just not going to happen. There may be some things like that where it is kind of like, it is almost like stocking roles. If the following [ Indiscernible ] is not true, stop and go spend your time doing something more useful.
I actually do think there is a value for having framework and then really thinking about what things in what buckets you could advance and really what your contribution could be. I do think that -- I do appreciate the point about some of these things I think -- can be implemented in lots of different ways. So example I used to use a lot is it may be important to have a system of reminders to make sure that people get certain things, so what is the system that my dentist uses? Every time I come to the dentist, I fill out a postcard which goes in a shoebox and gets mailed to me in a few weeks ahead of when I need to make an appointment. Our vet do something like that too. It is a postcard in a shoebox. It is not electronic. There is nothing electronic about it. And is actually quite effective and they follow. Some of it is what is the essence of the thing and not be how does it actually get done. I think helping people sort of not think that they have to have aerospace technology to make some things happen could be quite helpful in the meeting. It may be useful. I just wanted to put a plug in, and I think a lot of people have said this. For people with -- patients with multiple chronic conditions, is a worthwhile publishing to focus on. The system -- the evidence system and the delivery system are not designed today to deal very effectively with them so let's start there. it is not like we already know exactly what is do so there is a lot of opportunity for learning, and I think everybody can make a contribution. It is something that if we do it right by its nature it is very patient centered. The journey has to start with what is important for you and how do we figure out how to make that the guiding principle behind what were then trying to organize care delivery to do. To me, that would be a really important contribution. Some of these other things that you are interested in will come along with that, but I think it really is kind of -- to me, it is the best tracer condition on the healthcare system in kind of what the challenges are today that I think are basing both positions in care delivery teams and patients. It touches the social [ Indiscernible ] spaces as well as the direct care delivery space. That could give you in a funny way a focus. I appreciate Judds point about it might be simpler to start someplace else. Maybe you have a few easier things that are training wheels along the way, but I think even thinking about joint replacement surgery in the context of the patient with multiple chronic conditions is different than to replacement in a healthy 45-year-old weekend warrior or something like that. Anyway, Naomi.
I was listening to everybody talk and I tried to remember what my comments were. I think one of the things that I am having a hard time -- I thought I knew what help -- help learning -- learning health systems are. I think what I am struggling now with is the fact that -- I think it has to do with the fact that there are so much variability in health systems. I am working at the VA and they say well if you have seen one VA medical center, [ Indiscernible ] -- when I was working at CMS and we were working with Medicaid, we used to say it's you seen one stick, using one state Medicaid. I think there is so much variability across all the health systems that when you say well what are the core characteristics? What is the framework for a learning health system? I think it depends. As I mentioned, at the VA, we send [ Indiscernible ] -- we call them Quince. We don't really ask them. We ask them well what would be area Oka speed for your center of excellence? That is based on the medical center's needs, and it varies across regions. When you take a look at the focus, the area of focus on the centers of excellence that we fund, they are very all over the place because they are reflective of the needs of the veterans in that area. And reflective of the needs of the medical center there. so I am trying to grasp the concept of okay, we all believe in this learning health system, but I think they very all over the place. It also depends -- we talked about shall we focus in on the low hanging fruit texture we focus on one narrow condition, or should we work on something as complex as for other chronic diseases, patients and medical homes I found that it depends on help system. What is it that they are in most need of wax
I don't know if I am country getting any more than all of you have mentioned, but maybe finding out what are the core characteristics? What is the framework for the learning health system? That might very. I think Mary, you mentioned what is the definition because there are so many because of people's perceptions are so different. One might see it as process learning, but another might see it as something totally different.
You remind me of some work that was done with real people to test the term evidence-based medicine to which the response was isn't that what they already do? To the point about the presentational thing, thinking about failure to learn health systems for what the opposite is. It is an interesting point about the ability to articulate the next level. Sometimes, there is an advantage to talking about angst were everybody thinks they understand what you mean, and it is what they're already doing. So everybody feels included. And then sometimes, it is worth testing the degree to which the understanding is. There is enough of a shared understanding that you are actually going to get traction which I think is part of what you are going for, and I think it is appealing in that I can see how the our portfolio can kind of fit together in a nice way under that rubric. It is a reminder that there might be more to do, a little bit of testing outside the rooms of folks who pretty much think we know what you are talking about but even we [ Indiscernible ] pushed a couple levels down.
Someone raised to that with me recently and said well on the doctors already reading journals and all of these help systems? Are they learning X I said well, I mostly pile might up these days. But it is a very legitimate issue. I think we do have in mind this notion I think was something captured in something you said what -- a while ago in which was what has evolved because of the of billability -- availability of data on populations and so forth is the capacity to generate data, the systematically adopts data that is internally generated, and then that issue of the systematically making sure that that evidence, if you will, is made available to all the providers who are working at the interface of patient care. That is really different than the sporadic, random things that I might have read about in this week's New England Journal or if I picked up last month or things like that. That is really what we are trying to get at is this higher level of organizational capacity to harvest the information that can really do patient care at a higher quality and super way. I think some of what we talked about also is can we develop indicators of that that demonstrate progress towards that is probably an important part of our work. I guess the main point I take from your comment because it is something else I think about with regard to grant or developing something that we can communicate to others about this. We have to be very careful if we just say learning health care system. The general public are going to go well what is the opposite of that? Really help systems that are unlearning? Some people would say would be a good thing also, I understand. So I do think it is important concept to be able to try to get at.
I also want to -- Charlie, I appreciated were challenging us to make sure that this isn't just we are doing research about these things. I can't emphasize enough our desire to do this anyway that is cocreative, that is being done in conjunction with the end-user. Something -- if I could just maybe teared up a little bit more to see if there are additional ideas, I think I heard a little bit more around the so far. If we were going to do that, any particular strategy about -- we talked a little bit in general about the kinds of health systems that might be right in a sense for us to work with, but I am also trying to think about ways of how we can recognizing we are a public organization. How do we do this in a way that really thinks about the right kinds of partners to have for this and if we could even identify the organizations -- is it what we suggested a little bit here? Do we want to be getting into the chief medical officers, the vice president of nursing, the head of the safety office. All of the above. I guess I conceptualize a lot of what we are trying to do is to work at that interface of clinical operations and being able to figure out how evidence and research has to interface with them. I just want to make sure people are going well obviously, it should be working with so-and-so. They are writing the space. I just want to make sure that if you have thoughts or ideas about that because maybe I should start by saying that there are several of you on this Nack who give us connections to health systems, but we don't have a lot of other regular avenues that we are getting input from the help system and the provider community. We have particular projects that might come up, but we don't really have that. I have been struck by that. I have been trying to think about how to foster that. If you have notions about how it AHRQ might strategically think about pursuing that, that will be helpful to hear about.
At the risk of -- I was trying to decide whether I should do this as a side comment. I guess we will find out later. The other thing that strikes me we talk about different levels of system in organization is kind of at the heart of all of these are the challenge of lifelong learning and the part of physicians and clinical teams. There is a lot that is in flight right now about how we figure out who is effectively, and I will just say in the physician community, on a lifelong journey -- a learning journey. So there might be an interesting opportunity for some of the boards to think about reaching beyond systems into -- that will cut across systems -- in two ways in which some of these frameworks could be helpful with respect. Both thinking about all of the concepts related to adult learning, but also ways in which people on the front lines could be engaged in learning in a way that doesn't deal like either of my God, one more thing. Or irony know everything I need to know, thank you very much. Or something else. It is actually that there is synergy that could be developed in that we are trying to figure out how to keep people learning and how to do that effectively and reaching beyond systems is the only way to get there. So I don't know. When I was thinking about that, the stroke example, the three things I wrote down that I felt were missing were patients. I would sort of say, I think increasingly, we are recognizing that we shouldn't do any of this work without having really engaged patients with us on the journey because I think we need to increasingly get them to help us to find what a good outcome looks like to them. And to be able to measure that systematically. And then the frontline clinicians was the other missing group. And in the third thing was just a method which is really to kind of emphasized the importance, and I know you guys know this [ Indiscernible ] method, there are a lot of [ Indiscernible ] and other qualitative methods that help get out the stuff that are pretty hard to get at in this qualitative data analysis that is that conceptual piece that is important. Anyway, I would think of some ways to reach into the world of practicing physicians.
Beth, I think that is a great idea. ABI and Mark process emphasizes individual or perhaps small practice improvement cycles, but maybe they could be [ Indiscernible ] to look at and a broader system improvement cycles. And then in reference to your question, had he been to the American medical group practice Association? And its subcomponent GB Council of accountable position groups. They are your prime suspects.
I was just going to say I don't know where the link would happen, but I think there is a lot of great literature and learning from the practice aced research networks about engaging clinicians and patients increasingly in partnering with researchers. And so thinking of PVR and as these laboratories and learning help systems AHRQ has a long history with the PVR and so maybe there are some learnings there as far as -- I think many clinicians at the Alain once at least understand that it is one more thing but that it helps with their lifelong learning and so they tend to get something out of the partnership as well as the researchers. I know that has always been something that PBRN have focused on learning more. How do you make it [ Indiscernible ] and folks on the frontline at the same time learning together makes it work. We might not need to re-create that will. We might learn a lot from PBRN.
This is jumping back to an earlier topic, but I will be brief. I just came back from the society of medical decision-making meeting, and I saw a wonderful session that was organized by the VA on something called that's like me. They are using the very rich data from the VA and you can put in this person has a dead, diabetes, all those things and get a cohort of people and look at outcomes. There are methodological issues but I was very excited looking at that. It is a mechanism to be able to understand better how more complex people function. I thought that was really exciting work. I just want to remind people that we have a special emphasis notice out on getting [ Indiscernible ] outcomes and conceptual data into the ER. If you know people who are doing work in that area, please ask them to think about submitting stuff to us.
The one cautionary note I have about getting something into be HR is if I had a good idea today, in 2020, it might get onto the list of our IT folks for implementation. So I know that people had this wondrous idea of the EHR as the magic solution to all of our woes in healthcare, but the management of those systems and even sort of what seemed to be simple fixes to improve them, it is really striking to me how hard it is to get those implemented. So it is just a cautionary note. I told you about the shuttle projects that I was handed. At least half of those can't go anywhere because not being able to get IT to agree to put them on the list of implementation priorities. Then I will just say that we have a similar thing with patient reported outcomes were we actually have adopted a number of them, but the way in which they live inside the electronic health record is both quite varied and not always all that usable by our clinicians. The simplest ways to put them in turn out to be not very helpful from a clinical management perspective. I think we have to kind of -- it may be that we are moving to some new models in terms of information or maybe EHR is shorthand, that's where the ideas go to die. If we think that the solution is somehow putting it into the print -- EHR. The other thing is I saw some research presented a few months ago where folks had talked to patients about what they were and weren't willing to have about themselves placed in their electronic health record. Just a reminder that also with some of these things particularly in the social determinants area, there is a sensitivity to what the implications of that are for actually making situations worse and not better. So just some sensitivity to exploring whether patients would find that desirable. Mary.
I just wanted to agree with Beth -- that notion of who our key stakeholders -- I think [ Indiscernible ] representatives that you mentioned are of course important, but we are doing a number of help system visit. I could not stress more the importance of making sure that you have the frontline clinical team represented whoever they might be. In many hospitals today, we have patient family advisory councils. They are largely in hospitals. They have not yet moved into the community or connected with primary care practices, so one way to tap into that -- the national Academy of medicine has a network of patient Emily advisors -- patient family advisors who are leaders of these councils, and that might be a good source went to identify the help systems of communities where you are going to work. The only other group, I think you may have mentioned to this, I think it there are board members particularly those in health systems who are leading the newer quality agendas, I think it would be terrific to make sure that they are included. I will just reiterate. It there is any opportunity to engage kind of community-based organizations, even if you are focused on integrated health systems that have hospitals and postacute sectors to really think about community-based organizations, social service leaders as part of the stakeholder group. There is very little that is going to be done into dancing and learning help system that does not really begin to think about the connection between health and social services.
Anyway, I want to thank the members of the neck, both those in the room and on the phone. And also, Andy and Sharon and Arlene and Francis and David for your presentations. The AHRQ staff in audience who attended in person and those who watched on the webcast, I did get a report at lunch from somebody who was watching on the webcast saying that the quality of the sound and everything was really quite good. So that is always good to hear that technology is in our winter. -- In our corner. I want to thank all of you who are retiring for your service. As I said, we know where you live and we know how to find you. Particularly, many of you have made really important contributions, and it is hard to imagine we will just let you alone. So please return our calls. Let me just to see if anybody had any kind of parting thoughts, final comments that they want to share or leave us with, including ideas for future meetings. I think Andy proposed one idea that might be coming out that -- but we are certainly open to other ideas people might have. I have one more thing.
The next NAC meeting is scheduled for Friday, March 24, and it will be in the same building .
First of all, I just want to add my thanks to Beth and all of you for engaging --
the next meeting is actually at the Humphrey building. I have a discrepancy in information.
So I just wanted to thank Beth and thank the NAC for all of your great input today and also for my colleagues here from AHRQ for helping me to present an address your questions. I think we have gotten a lot of helpful advice. I feel certainly a sense of excitement about our trying to build towards this focus in our work. I also think we heard a lot of really good useful suggestions about ways that we need to think about how to prioritize this and some constructive steps that we can take to work with partners around it. I also want to mention just because some of our centers in the simple today and some of the work, but I do think that assuming we do engage in this area of learning help system, I think it also will involve all aspects of AHRQ. For example, we didn't hear from [ Indiscernible ] today who is the center that oversees all of our safety related programs. I think there is a tremendous amount of opportunity to align the work we have been doing in safety and thinking about engaging with help systems in a similar way to try to build more capacity within help systems to take on the challenging work of improving safety in the same ways that we were talking about improving quality in much of our discussions today. I see great opportunities there. I also see that in [ Indiscernible ], we have a lot of expertise in terms of analytic capabilities that will help us as we develop some of the data systems that were talking about here. I do see this as a really broad-based focus area for us that really ties into all aspects of AHRQ.
I appreciated that reminding you about if you do have thoughts about what might be an important area of focus for upcoming meetings, something I heard a lot of actually in your comments that maybe was a resonating with something that I had suggested earlier, but you brought more depth to it. I think we have been interested internally in developing ways of measuring impact of our work. I think also as a came up in the discussion today, as we think about training people within learning help systems and so forth, figuring out how do we develop metrics of what they're producing as a way of measuring NAC as a way that they can communicate that both back to us but also to the organizations to show impact that they are having there. we certainly had a robust discussion around while is this going to be research papers or is it not going to be research papers but I think part of what always happens is when you are not counting research papers, it is like okay, what you count, and how do you do that text that is something we are spending a fair bit of time thinking about here. I have raised in other settings and I will raise with all of you that if you have examples of ways that you are doing this in your own environment, we are very interested in that. We think this is something that is very important and that AHRQ could play and organizing role in helping to standardize again ways that this is done. So this is something that we are thinking about maybe for an upcoming one of our meetings to engage or help with. We are developing some of our own up luminary thoughts on this, and we would welcome your help on that. So I think unless my colleagues have anything else to add more say, I just want to say thank you for coming to AHRQ. It is so helpful to have the connections of all of you to keep us grounded in what we are doing. And to really test out and Our ideas and get all the great advice. So thank you so much for your commitment to AHRQ and for those who are staying on, we look forward to seeing you at the next meeting. For those who are departing, as Beth says, will be in touch. We absolutely do you as part of the community so thank you so much. I think I am ready to say I am done.
I am done. Thank you again. Keep those [ Indiscernible ] and numbers coming.
Thank you on the phone.
Thank you. [ Indiscernible -- multiple speakers ].
[ Event concluded ]