Event ID: 2663658
Event Started: 7/24/2015 8:25:59 AM ET
Please stand by for real-time captions.

I am calling this meeting to order. I think those of us who were gathered last night had a chance [ Indiscernible - low volume ] I am very pleased that she is here [ Indiscernible - low volume ] Sherry Lang --

-- There are two sherries. -- Sherrys. She is attending for Patrick Conway. Want to welcome the rest of you as well and everyone who is participating in the meeting and the attendees and those who are viewing on the webcast. This is being broadcast around the world, as a reminder.

[ Laughter ]

If you need transportation after the meeting please let the registration desk no -- know. There will be two public comment periods. One is at 1145 -- 11:45 and the other is at 2:15. We are going to go around the room now. Please make sure you use the microphone for the people on the webcast.

I am Jamie Zimmerman and I am the designated official for an icy.

[ Indiscernible - low volume ]

Jeff Weissberg at the Institute -- [ Indiscernible ]

[ Indiscernible - Distorted Audio ]

Chevrolet, -- Sherry lying.

-- Lang.

David Ballard, chief quality officer for Baylor Scott and White health in Texas.

Andrea gals are -- Glezer.

[ Indiscernible ] health services advisory group, Medicare improvement organization.

[ Indiscernible ]

Sherry Davidson, vice President on the -- [ Indiscernible ]

Rick [ Name Indiscenible ] director of HRQ -- director Agency for Healthcare Research and Quality . [ Indiscernible - low volume ]

[ Laughter ]

Since there is no budget for food, I took my posttest -- post tax honorarium to Cosco. We can get this set up at the break.

[ Laughter ]

I bought a bunch for my family and I said why not is it looked so good.

[ Laughter ]

Thank you. Can I check to see if we have any members of the NAC on the phone.

[ Silence ]

The first order of business -- I hope you got a chance to look at the draft minutes from our last meeting [ Indiscernible - Coughing ] There is a copy in the folder. I want to see if anyone has any changes or edit that they want to recommend -- edits that they want to recommend. I am hearing on. I will entertain a motion to approve the minutes.

I second the motion.

All in favor --

Aye.

Are there any oppose?

[ Silence ]

Thank you Pat. It is great to see everyone. Welcome to Jen at her first meeting. Thank you for your service. I am going to spend a few minutes on the budget which is a concern to many people, myself included. I will spend some time on general up dates, what we have been up to in the last four months. I am looking for input throughout. We were -- are going to return to a couple of items that were on the agenda last meeting that we didn't get to. I am talking about our work to extend patient safety beyond the focus of the hospital, and also some questions about [ Indiscernible - Coughing ] In the area of learning healthcare systems. Let me start with the budget this slide shows the history of our budget over the last 10 years. You can see the big influx of money in ARRA , B stimulus bill. Other than that it was relatively flat funding over the last six years, with some increase with the funds coming from the trust fund -- PCORTF trust fund. You can see it's quite flat over the last five years or so. The budget -- of the funds in 2015 -- about $200 million of the activity is in grants. The rest is about 70+ million dollars in program support that is paying salaries and electricity and things like that. There is money and contracts as well, a big part of which is the medical expenditure panel surgery and a variety of contract work. A large part of what we do is in grants.

There is often focus on the investigator initiatives such as the blue slice of pie you see here, the investigator initiated research. This is pretty much open-ended -- anything and health send -- health services research is eligible for funding -- that slice also funds some of our training activities as well as small conference activities, dissertation grants, -- are ones -- R-01s and R03s. These are estimates for 2015. A large slice of the pie is the PCORTF trust fund grants. In a couple of slides we will address that. There are a variety of efforts to disseminate patient centered outcomes research as well as training PCORTF researchers. A large part of what we do is grant activity and the purpose of showing the slide is that sometimes when people look at are granted think about only the investigator initiative and only $46 million was invested in initiated grants. There is also the HSR grants as well. The proposal for FY 16 was a proposal for 470 my know you and dollars -- $479 million which was an increase of four point -- $14.3 million from FY 15. [ Indiscernible - low volume ] An estimates of $115.6 million from the patient centered outcomes research trust fund, a slight increase from the FY 15 level. It is pretty much level for the grant part. This says $112.5 million which is just the trust fund part. This was a large slice and estimated at a similar level in 2016 to what we have been doing in 2015. There is a slight increase in overall funding, relatively close to status quo. That was the President's proposal. I am sure that everyone has seen -- we sent out an email to all of you a while ago -- be House Appropriations Committee has voted on an appropriations bill that moves all appropriated funding from AHRQ. The Senate Appropriations Committee has voted on a bill that would reduce AHRQ -- ARC appropriated funding by 5%. The Senate -- the 35% cut does set aside $45.9 million for investigator initiated research. Neither of these bills have been brought to the floor, either the House or the Senate. [ Indiscernible - Coughing ] Reading the papers, there is a fair amount of discussion of the likelihood of a continuing resolution at least four October 1, whether it is short-term or becomes [ Indiscernible - Coughing ] Is not clear. We have a strong support from the administration. The OMB director, Sean Donovan published a blog soon after the House Appropriations Committee took action -- saying that the administration thought a lot of what we see in the bill was ill-conceived. AHRQ was mentioned prominently in that blog. We are not the only targets in the House bill. The House bill and the Senate bill each remove all funding for the Affordable Care Act. The House bill also has large cuts in the Department of Education, National Labor Relations Board, other administrative functions for CMS. We are not alone. As I said, Secretary Donovan expressed very strong support for the agency, Secretary Burwell, in an interview -- the ousts -- the house action came just at the time of the Kennedy-Burwell ruling. The secretary was interviewed shortly after that. The first part was about Kennedy-Burwell and the second part was about appropriations. The first thing that she said about what was wrong with the appropriations bill is that it defunded AHRQ . We have strong support within the administration and we have seen an -- this is a cause for concern. It creates uncertainty. We are continuing to be hiring people and it is a difficult environment to higher in. -- hire in. Many of you have seen this before. It has created concern. I am one of the newer folk, for newer folks it is [ Background Noise ] That is the appropriations situation as we sit here today.

Do you have a sense where the opposition is coming from? Back when the agency started and they came out with the first set of clinical guidelines which were about the importance of conservative management of back pain and limiting surgery to those clinical situations where it has been shown to be helpful, as opposed to being harmful or having no impact, we knew where it was coming from. I haven't been able to -- the usual suspects seem to be supportive.

I would be interested in input from all of you. The broader context there is the budget caps from the sequester bill and they need in the [ Indiscernible ] bill to cut $3.8 billion from FY 15 levels. That is a context -- they need $3.8 billion from somewhere. It doesn't necessarily mean that it should come from AHRQ. That is a context in which we are working. As you said in 1995, when they were having big attacks on the agency, the upset was about -- we shouldn't be in the guidelines development is this. I have not heard -- development business. I have not heard these similar kinds of things -- we don't like what you are doing. What I have mostly heard is, we think you are doing important work but we are not sure that there needs to be a separate agency to do this and perhaps other agencies can do that. I would be happy to provide a response to that. I look forward to any of your questions first. I also would love any of your thoughts, if you have heard anything about why.

You guys are minutes up situation because the agency isn't allowed to lobby and advocate. You can't solicit support from people who you know are supportive. I am wondering in particular, given the situation in which you are describing, whether the stakeholders that would have general credibility with the people who are concerned with the challenges of balancing the budget have made their opinions known. I am thinking like the insurance industry, hospital industry, nursing professional societies, in patient advocacy groups. Is this just the beginning of a process and the responses are trickling in?

There has been -- the answer is yes.

[ Laughter ]

[ Overlapping speakers ] opinions known and there may be other work going on to try to make those opinions known more forcefully. At a time when we are collectively, deeply involved in delivery system [ Indiscernible ] and trying to figure out --

-- How it works --

-- How to improve the delivery of care, which is the work that the agency is fundamentally involved in, seems like a particularly inopportune time.

The agencies -- it's about 1/100 of a percent -- .001% of all of the dollars. It seems odd to try to make the -- it seems to be important to try to make the other 99.9% more efficient.

I am wondering, all around this table, we all care anonymously about this work. I am wondering if you could share from your perspective what it would mean -- what we would lose if AHRQ was a limitation -- was eliminated or even if AHRQ 's budget was cut by 35%.

If AHRQ were eliminated, we would lose the ability to look for evidence of how to make healthcare safer. The work funded over the last 15 years has produced the evidence about how to get central line infections 20 and had to reduce -- to 0 and how to reduce catheter line infections. We have in developing the methods on how to implement this across health systems around the country. With our partners in the federal government and in the hospital industry, be results of that work have been phenomenal. The report that we issued in December that showed that from 2010 to 2013 the rate of adverse events in hospitals declined from 145 from 145,000 -- 145 per 1000 145 per 1002 101 -- per 1000 to 101 per 1000. There is no other part of the federal government or private sector that is funding the work to figure this out. We would lose the ability to generate new evidence about how to improve the quality of care and lose investment in measuring -- how to measure quality. We talked in the last meeting about paying for value and the difficulties of that. The agency developed the cops -- CAHPS . We will talk about this afternoon, assessing patient experience in hospitals in ambulatory care and a whole variety of settings. This is a fundamental part of trying to improve the experience that patients have. We would lose the investment -- the ability to understand what healthcare is delivered to whom. The information that comes out of the medical panel survey and the health cost utilization project -- survey is fundamental to much of our ability to produce evidence about how to improve quality -- in the NEPS case an estimate of what will happen if we change policy around employer contributions are provide subsidized insurance -- [ Indiscernible ] -- [ Technical Difficulties ]

Showing the limits of my power.

[ Laughter ]

The one thing that I intend to do in government -- I am trying to get that change to "You were" -- " fewer" It should not be "Less" [ Laughter ] MEPS is fundamental to estimating what would happen -- NEPS is fundamental to estimating what would happen when we go to centralized healthcare. Amazingly we have this estimate pretty close to write and we were able to do that because of maps. -- NEPS. We would lose the investment in research to figure out how health IT can be made -- can be used to to improve safety and quality. Many areas in which we would not be able to be producing the evidence in making sure that it is understood and used about how to improve quality, safety, accessibility and affordability, and the equity in delivery of healthcare. If we were to have the 35% cut, we wouldn't lose all of that. We would lose pretty much all new investment in research around patient safety. NEPS would take about eight $30 million cut and be decimated. We would have to change that survey and tiredly and it would be much less valuable. We would not be able to fund CAHPS or HCUP or the quality indicators that are used by CMS and more importantly my many hospitals to try to measure and improve the quality of care. There would be a reduction in funding for HIG grants -- HIT grants. In the 2016 budget we have an exciting initiative to help people with multiple chronic conditions. We would certainly not be able to do that. It would be a very large reduction in our ability to produce evidence and improve equality.

I was led to say all of that reinforces the central role that [ Indiscernible ] [ Background Noise ] Has played. Anticipating all that we are confronting in moving forward, all of what you have described, we need rapid movement in production of evidence in all of these areas to continue to grow. They are essential to what we are talking about, in terms of health system redesigned and achieving what we -- redesign and achieving what we have set out to do. Thank you.

Mac I wanted to add to what -- [ Indiscernible - speaker too far from microphone ] I wanted to add to what Richard has set. We are concerned about all of this but we are also concerned that the message will get out two new investigators and the investigator initiative work is the generator of new ideas that leads to the investment to make healthcare -- healthcare safer and improve quality. What we don't want is for new investigators or any investigators to not submit good proposals to us because of any uncertainty. Any suggestion that you how -- have on how to get that word out in the midst of this would be welcome. That would be too bet if we didn't get the new idea generation that is so valuable to our mission.

This is actually the first time I had heard that there was some protection for that, which is interesting. Someone got to someone. Andrea, you are next.

For all of the reasons to which Rick so passionately spoke, I'm going to pile on to what Mary just said -- I would say who better than AHRQ to develop translational evidence-based guidelines. Quite honestly we have the national quality forum that takes measures from a hodgepodge -- I sit on those committees. It is great work but in most cases it is very singular and I can't really see what we do in those committees translating to improving outcomes in the clinical setting. I think it is critically important that AHRQ continues its work.

I was at Emory when my Congress current -- Congressman had concerns about [ Indiscernible - low volume ] When Sandy and I served on the healthcare technology section we were young man at the time.

[ Laughter ] You are still young. [ Laughter ] We thought we knew what we were doing back then. [ Laughter ]

At that time it was pretty clear who had the concerns and what the concerns were. I had this great discussion with the staff about -- this agency doesn't want to fund research and wants to tell doctors what to do. I had a chance to talk with them about some of the work [ Indiscernible - Coughing ] Clinical practice for doctors and other providers. Maybe this does exist. What I think would be helpful is some kind of communication device. Maybe it is out there and I haven't seen it yet. Maybe you can't put it together but maybe some other group can -- that could be some common document that we can reference and sent to our Congressman or Sen.'s. I don't know what you're limitations are in that regard.

We must all be thinking along the same lines. Rick as your -- as you alluded to, AHRQ is a very splendored thing. People have an idea about what they do at [ Indiscernible - low volume ] Clinical science and cutting research but I don't know that AHRQ has that soundbite that is understood on the hill. Maybe if the key -- could be framed in terms of almost things that you mention, improving health care for Americans at every stage of life. It would get that second-level translation from clinical science to health systems a little clearer in people's heads.

Sandy?

I want to underscore what Jeb was just saying. I was having an extended conversation for a national newspaper yesterday. Questions I was being asked -- he was approaching it from the question of clinical science -- highlighted the issue, the basic sciences fundamentally important part that is where the breakers will come from. At his attitude -- there is attitude out there that if you build it it -- build it, they will come. He was asking questions of not doing enough evaluated research or systems designed to know how to use things appropriately. I think we haven't done a good job as a community in terms of articulating that.. These fundamental discoveries that ultimately will derive from biological sciences coming out of NIH in particular and other industry and other places -- we need to know how to use them. It's not a question of good or bad. We always have clinical people -- it's not question of good or bad. It's more of a question of how good and under what conditions. How do you organize that care? I can't think of a major policy that either the legislative or executive branches limited -- implemented that has not had its roots in programs that were funded by this agency. I think we need to figure out how to do a better job communicating that, recognizing that the NIH public relations budget is probably larger than the staff budget.

[ Laughter ]

These are all excellent points. Sandy, you asked earlier, what is the source of this. The context of the budget caps but also the sense of what we do is important but other agencies do it. As others have suggested, some people don't know what we do. I think it is these two things.

I think there was a lot of confusion on the hill in particular, some things that the agency does, other people either get or take credit for. What I was thinking to my you have to tell me if this is legitimate. I think this counsel might be able to help the staff in terms of providing feedback about educational information that clearly communicates what the agency's mission is and what contributions it makes and why it would be important -- generally going forward for the next few years. I think other groups you have working with you, stakeholders groups and stuff, we might be able to -- we provide feedback. You decide whether to listen to what are not. [ Laughter ] We don't even agree with ourselves most of the time. And I am talking about a personal level, not on a group level. We all come at it a little differently. There might be a couple of good ideas a come forward.

I am confident that would be very helpful. I want to try to respond to the other agencies do what we do comment. NIH, as I think everyone knows, focuses on discovery, try to figure out what works -- trying to figure out what works. We focus on the translation and to make sure that what works actually gets implemented. We all know -- Beth did the ground breaking work showing that half of evidence-based care doesn't get delivered. That is still the case. Barely more than half of people with high blood pressure have a controlled. We know how to fix it but we haven't been able to do that. NIH doesn't work on that. That is what we work on. We are sometimes -- one of the criticisms is -- CMI is doing everything that you guys are doing around delivery system reform. You may want to comment.

For 20 times as much money

[ Laughter ].

CMMI is doing fantastic work in changing the financial incentives that writers face -- providers face. As you all know the question for delivery systems is, what do we knew now.

Then a miracle happens. [ Laughter ]

We have great incentives but what do we do? What HRQ does don't what AHRQ does is providing answers to those questions. We are the only agency that is focused -- what AHRQ does is provide answers to those question. We are the only agency that is focused on that. We want to try to make sure that the evidence is used, dissemination, very different from other agencies. There's lots of contravention work -- complementary work. We work closely with people at NIH and ONC and others on this. I think sometimes complementary work is mistaken for duplicative work.

As being one of those that was here when this movie played last, the other factor back in 1996 was healthcare reform. The agency was being accused of being behind Clinton health care. Some of this is you are being a surrogate with bottles -- about battles with ACA. This probably doesn't help you. I think the story about healthcare associated infections is really a great one to push because what AHRQ did that know whether agency did such as CMMI or NIH was followed with through from early proof of intervention to taking it to scale. You could probably calculate the economic return on that, relative to the investment, in that research, but even relative to AHRQ 's entire budget. To make something happen it requires the focus on getting something to scale and building the measures and everything so that a health system -- so something isn't happening in just one hospital but across all of Michigan and then also across all of the states. You could put dollar costs related to that. Just as a case study, that might resonate with people in a way that survey data and all of the other essential stuff that AHRQ does , does not.

I would echo all but has been said thus far but also add to it. I have a different perspective. What is now required and what has been proposed rulemaking, including quality reporting for all care settings that actually builds on the work and work products of the agency. One classic example is the use of CAHPS that we have tried to use across all care settings. This is critical as we are trying to reform the delivery system so that we get right. Not only that but to amplify that, not only is it in the inpatient quality reporting programs but it is part of value-based purchasing now. How do we continue to know that we are working -- continuing to work in the right direction. The impact of that is -- the collateral impact is profound, touching on every care setting that patients and families cycle through -- that is one element that we have already proposed and to pull the rug out from under all of that would be quite devastating. I would also add, echoing what David has stated, be evidence icons and we have built on for technical assistance through the QIOs and hospital engagement networks, all of that is actually informed and refined on the basis of the agency's work. But it is not just about the technical assistance. Through our collaborations with your agency staff and leadership, we have come to really learn the value and importance of the culture of safety end of changing and improving. I think that part also is really built upon the work of the agency. The last point I would like to make is something that has been alluded to but as a practicing geriatrician I think it is important in the clinic. What matters is what we know and what we do not know. We will not be able to use information if it is not the point of care and we know that for a fact because of the work of the agency. If you amplify that commonly classic example of that is what we know about the evidence and what is sufficient and what is not sufficient. That transferred -- that translates into cover policies. Just this week we had Medicare evidence development advisory committee meeting that built on the foundation for which -- on peripheral artery disease was the work of this agency. I think as we look at this, it is not -- the numbers will be astounding. The cost in dollars are important but it translates across and permeates across the care setting where it really matters to patients and family members, including at the point of care. Thank you for all of your work. That is all I have to say.

I want to build on that but from the employer perspective, from the commercial market. Commercials -- employers have been on the sidelines with the changing reforms, waiting to see the evidence that the new financial incentives will work. A lot of our members say, been there, done that in the 90s with primary care and moving to those kinds of managed-care components and caps. There waiting to see what comes out of this. They do want to pay for evidence-based benefits. They want to help employee -- there patients navigate the healthcare system. I don't think that was a long that's enough until we have the information that comes out of this agency. I also think that these are real lives and that is not something that you can discount.

I think the work that the agency is doing is incredibly important in terms of translating what we know and bringing it into the clinical setting. A couple of things, I guess I am very skeptical. I don't mean to be negative. I'm going to be negative about these people anyway. There is a real disconnect in Congress, I think, based on my own observations, between the medical research that is being done and the delivery of that care to patients. We all hear this all the time. We have the best medical system in the world. We have some of the best medicine and some of the best breakthroughs but we don't have the best delivery system for that we all know this. There is plenty of evidence to show that. Somehow that disconnect has to be addressed. Maybe one way is to start by having on the website, we are making healthcare better because -- and we can list some of those accomplishments that you have articulated. It is hard to get that out of what is on the website right now. I think that might be a place to start. The other thing is -- NIH is the sacred cow on Capitol Hill . They can do no wrong. They do wonderful research. There is also room for improvement there. They are not charged with translating that research into the medical system. I want to thank Sherri for bringing up the cat's surveys. I think that is an -- CAHPS surveys . I think that is extremely valuable especially from the patient perspective. There is nowhere else where that information is being explained. Is that care affordable and I things being explained in a clearly understandable way? I'm going to get on my soapbox. Congress thinks no one should get between you and your doctor unless it is them. [ Laughter ] The work that AHRQ does is about translating all of that important research and looking at the delivery system and making sure that people get what they need. Who was getting it? Was not getting it? How can we improve? Were the only people that do that. I think that is a big -- we are the only people that are doing that. I think that is very important.

I would echo much of what I have heard about making care better and saving lives and quality and safety and accessibility as well as affordability. I want to highlight the importance every single day of the US presented -- preventive [ Indiscernible ] also the work that has been done in the practice-based research networks on the ground in the real world touching communities, touching patients. I was approached recently by one of my colleagues to think about building on the tools that AHRQ has created. I was thinking back not too long ago where I had these 3 x 5 cards in my pocket of my whitecoat. Now I have it on my computer. For that to go away would be such a huge loss in primary care. We want to prevent this illness as well as high cost care. I wanted to bring up those two examples that I didn't here. I know you know about them but I just wanted to highlight those as well.

This is Paul [ Indiscernible - low volume ] I just joined.

That they can't see your tent card, I want to check and see if you wanted to say anything on this topic. Don't feel compelled?

I think he just joined so he doesn't know what the topic is. [ Laughter ]

[ Laughter ]

Free response. [ Laughter ]

Thank you all for your comments and suggestions. This is a useful conversation. I would say that [ Indiscernible ] help -- health is doing a lot of work on AHRQ's behalf . If you would be interested in connecting with them, that would make sense. That was a very useful discussion for me. I hope it was for you as well. There were some very excellent suggestions. Let me move on to the update part. I am delighted to report that our -- Arlene Bierman will be joining the agency on August 9 as a director for the Senate -- center for evidence and practice improvement. Arlene is a professor at the University of Toronto and has done very exciting work in quality improvement, creating a quality measurement system in Ontario. This is her second tour at AHRQ. She was here about a decade ago. I know David Myers who has been acting director of the center for about six months now is looking forward to Arlene's arrival as are the staffing center [ Indiscernible - Coughing ] We are excited and she has a tremendous amount of talent and energy, as I have mentioned before, I would remind folks, David has been wearing two hats. He is the chief medical officer at AHRQ but has been so consumed as the acting director of the center of the CMO job that is a job in the making. I am looking forward to David being full-time [ Indiscernible - low volume ]

Wasn't Arlene be driving force before the ambulatory care rentable conditions measures? That goes back quite a while. That is what I remember her for.

[ Laughter ] [ Indiscernible - papers rustling ] [ Overlapping speakers ]

Ginger, I don't know if you know her, may have a view on that. [ Name Indiscenible ] was key in the development of those as well. I am happy to say that in early October, October 4 through October 6 we will be holding our research conference which used to be called the annual research conference. Given the concerns about conferences for a while, there has been a two year hiatus. The conference will be held in Crystal city. Became is producing evidence and engaging partners to improve healthcare -- the theme is producing evidence and engaging partners to improve healthcare. [ Background Noise ] I think it will be an exciting couple of days. There will be some great plenary sessions and many research sessions. The conference is Monday and Tuesday morning, October, October 4 and October 5 -- before that weekend and then we will be holding a session with 1015 -- PCORI which is holding a conference on Wednesday and Thursday. Be Tuesday afternoon will be the joint sessions. I am hopeful -- the Tuesday afternoon will be the joint sessions. I am hopeful that you might be able to come to the conference. As I think you know, we have got four priorities at the agency around quality, safety, accessibility and affordability. [ Background Noise ] What are we doing around those four priorities? We have talked with you before, David Myers, a couple of times about the initiative to improve the ability of small and medium-sized practices to incorporate key COR evidence into their practice with a particular focus on improving cardiovascular risk factors. The new news is that we have actually awarded the grants. The last couple of times we were getting ratty and we were going to announce the funding opportunity. We have now awarded the grants to seven regional cooperatives around the country. We will be working with about five Mary -- 5000 primary care clinicians at the Oregon health and sciences University to conduct an overarching evaluation of the entire initiative. Each of these groups is doing outstanding work themselves that we will have the overarching evaluations as well. As we have discussed, it has dissemination -- very substantial dissemination components. We will be working with 5000 primary care clinicians taking care of about 8 million patients. There's also a strong evidence -- presence of producing evidence on what works. The cooperatives will have on-site practice facilitation, data, feedback, benchmarking, other approaches to try to make progress. The grants were awarded in May. The grantees are working on establishing the cooperatives and recruiting practices. The active intervention will begin later this year with a period of 18 months to 2 years. The data collection will be every three months. The main measures of success here will be improvement on. -- PQRS Measures and also the measure of the ability of this is to [ Indiscernible - Intermittent Audio ] As it emerges over three years for the grantees. And one that I am very excited about a looking forward to -- the other large investment that we have talked with you about before but we have new news on. We have announced awardees -- to fund three centers of excellence in [ Indiscernible ] in disseminating PCOR. This is focus now on small medium practices this initiative help Tuesday -- try to look to see how the work relates broadly to system performance in quality and resource use. The three grantees will be Dartmouth College, Elliott Fisher is the PI there, a collaboration with UC Berkeley with Steve there and the high-value healthcare collaborative that will be doing a fair amount of claims-based analysis, but also a national survey of healthcare organizations. This could be a very exciting project. The national Bureau of economics research, David Cutler is the PI. Might -- Mike Churno is working with that as well. The national healthcare initiatives -- national network for healthcare initiatives -- were put all the data together and tried to understand what systems are and what they are doing. RAND With Sheryl Sandberg collaborating with [ Indiscernible ] at Penn State -- this is reason only -- regionally focused for a deeper dive in regional -- into regional healthcare systems. We will be funding these grants in September and I look forward to these five-year grants. We are looking for a variety of intermediate products. We are not waiting for five years for the outcomes. There's quite a lot of investment needed.

This is Paul. I want to mention that I am part of the RAND team on this project and I am looking forward to it.

We look forward to your work on it Paul. Thank you.

[ Laughter ]

Says we missed -- since we met last, we have release the national healthcare quality and disparities report, QDR which used to be known as the QRDR. We have combine them now into one report. We have a session coming up after this in which Ernie Moy will be talking about these. These were huge reports. We are still producing virtually all of the material in the reports and they are being released on a rolling basis as web documents trying to get more focused on each of the chapters. I look forward to the discussion we will have on the QDRO -- QDR. This guide, and lamenting a state level quality improvement collaboration -- implementing a state level quality improvement collaboration -- the document that came out of work from MedNet that Arquette funded -- one of the kinds of evidence generation activities that we do to try to provide evidence about how you actually make improvements -- Jen brought up the activities of the -- and the importance of the US preventative service tax force -- task force. Said Celeste meeting they have finalize two recommendations on screening for thyroid dysfunction and speech among which delay. As I am sure everyone has noticed, they issued a draft recommendation in April on screening for breast cancer. There has been a lot of interest and concern about that recommendation. Much to the task forces -- task force's credit they have done a lot to try to improve communication. While there was a lot of concern relative to 2009 when he test force issued its recommendation. There is a much more reasoned environment and discussion professor to say that all of the discussion has been reasoned. In 2009 there were pickets outside of the agency. As I think you probably all know, the recommendation -- the draft recommendation that they released is very similar to the recommendation in 2009, recommending biannual screening for women, 50 to 74 as a B recommendation. And for women in their 40s, finding that mammography every two years can be effective and recommends that the decision to start screening should be an individualized one, after a discussion between a woman and her physician, recognizing the potential benefits and potential harms for women in their 40s. It is a C recommendation. That is currently rated -- created concern from a variety of folks and also support from a variety of folks. The organization a Carol is with this one of the organizations writing letters that this makes sense to us. Any questions so far?

The evidence-based practice centers have been continuing to be extremely active. You can see a variety of systematic reviews on this slide -- and technical briefs that have been released since we last met. We have also, since we last met, issued and intent to fund, in the first case, a funding opportunity -- an intent to fund, in the first case, I funding opportunity to [ Indiscernible ] sexing -- we are accepting applications for the learning network and we anticipate issuing an F00 -- two FOAs -- one to see how to take [ Indiscernible ] to scale and the second one to work on developing new CDS in places where there is good evidence about how to practice but not yet critical decision-support to support that evidence.

I will move now to work that we have been doing around producing evidence to make healthcare safer. We have talked about this already today. Jeff Brady talk to about it during the last meeting. In December we issued a report that showed that hospital care is 70% -- 17% safer then 2010 and 50,000 patient that's being prevented as a result. Resident Obama use that number in a speech he gave on the fifth anniversary of the a formal care act, in March, in which he was extolling the virtues of the ACA. He said one of the things the ACA had done was to lead to this result the 50,000 fewer deaths. The fact Checker at the Washington Post immediately got on this and said, what is going on. I don't know if any of you read Glenn Kessler buddy looks at what politicians --, but he looks at what politicians say and he says are they accurate or not. He rates statements as -- from 1 to 4, Pinocchio's. He gives someone two Pinocchio's -- Pinocchios. Any said this is Exley right. You gave it the Geppetto checkmark. -- This is actually right. He gave it the Geppetto checkmark.

[ Indiscernible ] [ Laughter ]

Thank you Sandy.

Getting into the grading mode [ Overlapping speakers ] [ Laughter ]

We also released a survey on patient safety culture in ambulatory settings. As Sherry said, an important part of what we had done is both point out the importance of creating a patient safety culture,.also creating a mechanism to measure patient safety culture that is used in many hospitals around the country and we are extending that to surgery centers. We have been working as part of a much broader HHS initiative around combating antibiotic resistant bacteria. The investment doubled in 2015 from the two -- 2014 level. Stewardship programs in nursing homes as well as promoting adoption of antibiotic stewardship in ambulatory settings as well. We have also been working on increasing the safety of medication use and issued a funding announcement that was titled as you can see on this slide. They were interested in projects designed for implementation of safety processes. [ Indiscernible - low volume ] This is in concert with the HHS national action plan. I will move on to work that we have been doing and producing evidence around increasing accessibility. We released about three weeks ago, results from the 2014 medical expenditure or panel survey from the insurance component. Many people who know what know what MEPS is have seen the information about household related issues. But this is also looking at what employers [ Indiscernible ] there was a lot of concern leading up to 2014 that the coverage expansions, the availability of subsidized coverage in the marketplaces, the imposition of a tax on insurers, be requirements in the small -- the requirements any small media markets for essential health benefits would cause a tremendous amount of disruption and problems on the employer-sponsored insurance market and potentially lead to increases in premiums. There were lots of stories prior to 2014 about premium spikes even in the employer market and concern that it would be to employers dropping coverage. Most of what we have seen in 2014 is that, notwithstanding all of that concern, 2014 looks pretty much like what was going on -- the changes from 2013 to 2014 are pretty much like what was going on in the previous five years, in some cases longer. This slide shows the percentage of employees working in establishments that offer health insurance. The national averages the middle blue bar. You can see that the percentage declined some from 2013 to 2014. It's pretty much in line with the decline that we have been seeing since the beginning of the recession -- much of that decline has been driven by Kleins in offering rates among -- declines in offering rates among small agencies. One of the strengths of the MEPS survey is because of the large sample size we can produce pretty precise estimates at the state level of offering rates -- premium rate. This map shows offering rates among small firms and you can see the light blue are states that are statistically significantly lower than the national average. The darker states are significantly higher than the national average. You can see a clear regional pattern.

[ Indiscernible - speaker too far from microphone ]

That is Alabama. It is a high state. I do not know why. I am surprised.

Vermont is a low state which is another surprise.

Vermont is an average state.

This is pretty hard to estimate at the state level. States that are not significantly different than the national average in our estimates, might actually be different. In some of the smaller states, it is hard to get precise information. Steve Cohen is here. Do have anything to add?

[ Indiscernible - speaker too far from microphone ] -- Do you have anything to add?

[ Indiscernible - speaker too far from microphone ]

There's obviously a lot of interest in premiums. This slide shows premiums for both [ Indiscernible ] as well as firms with fewer than 50 employees as well as greater than 50 employees. 2013 to 2014 increases, about 11 -- 12.7% for single employees, lower for family. It's pretty similar to the 2012 to 2013 increases. You probably can't see it here and we don't go back long enough. The period from -- [ Overlapping speakers ]

[ Indiscernible - speaker too far from microphone ]

-- 2011 to 2014 is a modest growth of about 3 1/2% -- 3.5% which is much lower than me 2008 to 2010 period. Some state detail on this, premiums are lower in the South and some of the mountain regions and higher in the mid-Atlantic and part of New England.

The last slide from this MEPS work is the slide that shows the fraction of establishments that self-insure it least one plan -- by firm size. You can see the largest firms on the top work it's over 80%. This is a boring slide. The point of it is, even know there is a lot of -- I can't take it a number of meetings I was at where people were talking about how much fight there was going to be -- flight there was going to be to self-insurance, there's not much change.

The last priority is the desire to provide [ Indiscernible - low volume ] Two days ago we produced a new tool called fast stats that allows users to have easy access to quite recent data from the HCUP project . As I think you probably know these data include the HCUP as a whole, 97 or 98% of all of the hospitals in the country. There is an existing [ Indiscernible ] called HCUP net in which people can do queries of the HCUP data. Fast stats is the second tool, more limited in some ways but very cool and need in other ways and allows trend analysis, easy comparisons across states and my point of view -- from my point of view very excitingly, access to much more recent data. We have HCUP data from 2014, from 17 states -- fast stats data from 2014 from seven states and you can use this to look at what happened to hospitalizations in states that have extended Medicare versus the states that don't. This is just one example of this. On the left is data from Colorado which expanded Medicaid and the bottom two lines is a number of hospitalizations that were covered by Medicaid in the age group that I can't read -- I think it says 18 to 64. That would be Medicaid adults. You can see a big increase. You can see the light blue line which is the number of hospitalizations for uninsured or self paid people. You can see a big decline in 2014. Colorado had partially expanded Medicaid prior to January 2014 which is likely what accounts for the diversion see that you see before 2014. Missouri on the right part of the slide, did not expand Medicaid. You don't see anything like that on that chart. Fast stats will also provide a number of hospitalizations for three diagnoses that are ambulatory care sensitive. I encourage people to play around with that if you are the kind of person that likes to play around with this kind of stuff. We will also be releasing a nationwide readmissions database in the fall. Again, I think this is an exciting extension of the HCUP data , the first nationwide database to be able to do work on readmissions, folks have been able to work with Medicare data for quite some time. But be able to do this more broadly would open up many new vistas. Any questions on any of this update stuff.?

It's a question. It's more trying it back -- does the agency track how this is used in ways -- [ Indiscernible - low volume ] [ Indiscernible - speaker too far from microphone ] --

We can share that document. -- This has been increasing over the last year. Only checked the number of publications -- we have a variety of metrics. I would be happy to share them with you. We have about 30 minutes which should work. Two areas -- that we are currently working on -- one area that we are work on -- working on and one area that we might work on -- in FY 15 -- were working on extending the patient safety work, which has primarily been focused on the inpatient setting. We have been working around the safety and nursing homes, particularly around reducing falls in pressure ulcers and trying to figure out who are the folks most at risk and what to do about that. West date -- less work today and way less that we put into the inpatient setting -- [ Indiscernible - low volume ]. Less work on safety in the ambulatory setting. We have been doing a lot to try to reduce diagnostic errors, much less in the inpatient setting. We are currently reviewing a row best -- a rope last set of responses now, saying that we are interested in working in this area. We have gotten quite a lot of interest and interesting ideas and responses. The FOA was open-ended. We are interested in this stuff, long-term-care, ambulatory settings. It talked about diagnostic errors it didn't provide much direction to researchers. I think as we are beginning this process, that is appropriate. I would be interested from all you, should we continue as we move forward in this open-ended approach or should we try to be more focused. If so, what should we be focused on?

[ Captioners Transitioning ]

I would look to others for a better response. Certainly we and others have done work on care transitions, which has been, you know, an area of pretty active focus. I think a lot of that work and care transitions has been focused more on the professionals that on the caregivers. So that is a good suggestion.

This is Anne Hendrick and I would add that what we see most and are recent homes or senior living care facilities is the lack of knowledge around high reliability principles and practices, lack of knowledge around team-based care and stimulating initiatives that embrace that. And as was just said empowering caregivers to really understand how to execute and implement in that way, that would absolutely improve the care of the readmissions we are seeing frankly from long-term care facilities and to the acute-care cycle.

[ Indiscernible-muffled speaker ]

[ laughter ]

Good morning everybody, briefly I will remind you of one tool we've had out for quite some time but still new enough that I think it is available for more focused uptake and that is the patient and family engagement guide focused on the hospital. That does not completely focus on the role of caregivers, but clearly they are part of it just is represented by the name of that resource.

And based on the success and interest in that tool we are following that with a similar tool for the outpatient setting that we clearly understand different challenges, and maybe a different format is needed for that resource, without is a project in development right now. Those are at least two examples with definite recognition of the importance of patients and families, you know, sort of navigating the system if you will.

I just wanted to put a word in for the diagnostic error piece. As you may know the national Academy of medicine, the group formerly known as the Institute of medicine, will be releasing a report sometime later this fall on diagnostic error. Much more broadly than outpatient settings. There is quite a focus on the research needs which are substantial in this area, and quite challenging. I think there will be some real opportunities to take a look at that, and I think the real need. I would sort of watch this space for that report. Because I think the research needs having sat on the committee are quite striking.

Absolutely. Yes.

[ laughter ]

We are a cofounder for that work. In addition to a lot of the foundational work, Mark is one of the leaders on that team as well, we have funded his work, so we have a pretty long track record and it is clearly a broad topic. We do not have a lot of insight yet into the report itself.

It is the nature of the process.

Yes but there is a coalition forming.

Good. I think Kevin, you had your card up.

I think it was Naomi.

Whatever, you work it out amongst yourselves.

There are two thoughts. I do want to echo Jed's comment of it. I think there is an area you need to include, I think it is great that you are including amatory and long-term care, but it is the role particularly of home care workers, California we call them in-home supportive services, I think there's a huge opportunity with people who have chronic illnesses who of a lot of dependency or disability, there is so much medication reconciliation, to empower and home type workers, and I know there have been some worker in California around this, to be an active part of the team and to accompany patients to the visits. Because they are the ones that know what is going on in the home. I do not know if you've done any inquiry in that area but I think that is a huge opportunity to look. Because there is so much of a mess. It is folks who often have some compromise and it is hard to know what is going on, the whole workforce has been way under empowered to pay more active role. That is my first one.

Very briefly, we absolutely agree, there is potential for more work. Another IOM report of us two years ago focus on information technology in the home and sort of highlighted that aspect. We had a grant of the past that focused on the assessment of the environment, sort of the human factors if you will, so we agree there is lots of potential. Although as Rick said the focus of this extension of patient safety to all healthcare settings is initially ambulatory and long-term care facilities, we are intentionally broadening adding compassing. In fact, a lot of the problems happen or exist between healthcare settings.

Maybe just caught as the home being under ambulatory care and maybe not having that get lost. Another point is that I think is great I like what you did with the hospital-based guide around patient and family engagement. I think that is a huge area, I think in amatory care, and in particular permit care, there is a lot of interest in that and so little understanding how to do it well and feasibly. It is one thing when you have hospitals that are well resourced that can staff and facilitate it, how do you do in a small clinic or small office practice? There is a whole body of empirical work to be done around how this one actually feasibly effectively will engage patients with an eye to safety among other reasons to be doing it? I think it is a huge opportunity to figure this out.

Thank you. Building on that, the care coordination work, is that being done just in the inpatient settings? It seems to me that this actually is an extension of care coordination. Many people, especially in ambulatory care where people are just visiting a number of different providers for various ailments that it is the patient or the family member who is often coordinating that care. And by extension then, in-home care. It seems to me that would be a place where there could be some better coordination and overlap, and if we could work on that it might be a place to start.

I just jump in? So AHRQ was funded under some centers of excellence to develop quality measures for children and I was affiliated with a center that was developing measures and care coordination for children with special health care needs. And one of the conclusions that we came to a network was that having looked at ways to do it in medical records, claims, and surveys, concluded that asking the people who were actually there on the front lines trying to do the care coordination made the most sense. So we are in the process of submitting that work both for publication, but also to NQF for potential endorsement. And I was at one of the reasons I mentioned that is, and it goes back to our earlier conversation, without tools to figure out what your baseline is it is very hard to figure out whether you improve care coordination and whether that is the mechanism by which you are getting improvements in care. There has been what I call the care coordination the Bermuda triangle because so many measurements to its have gone down in flames. We have certainly seen a lot of the reasons for that in doing this work.

I think a lot of this extends well to other populations analysis the population we were asked to study. I just want to endorse the comment but also recommend or recognized that AHRQ did some fundamental work that would help accelerate the work in the space.

We did some work in the Senate for evidence and practice improvement developed a compendium of care coordination measures much along this line of trying to understand what is available. It does not improve what is there but that them all in one place.

Let me frame this different, I think this focus on people living in the community and patient safety is central. Some of it may link to the work you are probably talking about with people who have multiple chronic conditions. Thinking about who is it the right population target for and who will benefit the most to promote the culture of safety and so on? On the workforce I could not agree more, the focus on who is the team now that has not been the team before, including the person and the caregivers, and California does have a great program going on right now. I think they are testing 6000 personal aid workers in building competencies and then integrating them into the team of people. So not specifically focus on patient safety, but as I think represent the opportunities heard

The only thing I would say is these times - kinds of interventions, especially people living through the complexity of a journey through multiple care and so on, are often better if they are tougher or multidimensional. So not just focusing on falls or pressure ulcers or medication, but really focused on if we do not get meds right we are can have falls. So thinking about the multidimensional approach to patient safety that enables you to simultaneously prevent delirium, prevent falls, and get people the right and appropriate meds. I think that is the direction we need to be focused on.

Sandy?

Just briefly, [ Indiscernible-speaker too distant from microphone ] I am very enthusiastic about this. Most health care is provided outside of the hospital. The challenge for the agency with a two limited budget is where to focus. Do you look at relativist reductions which will be people with chronic and serious diseases, or do you look at population benefits, which will be lower rates but they will affect more people overall?

I think in a stage like this probably want to earmark some for each. To have some areas of focus so you can have this. I think what Beth and Mary were saying from different perspectives, which I agree with, can you quote Mao in this form?

[ laughter ]

100 flowers bloom and 100 schools of thought prevail, I would like to encourage a broad range of ideas because I do not think any of us really know where these problems are. But they give out an area that I have done a lot of work and come diagnostic tests for example, we're taking often healthy people are doing the worst thing. We are taking healthy people and making them sick with the diagnostic errors that could be defined broadly. So you could think about that orthogonally or anywhere in between, but it would be good to be open to multiple ways of framing this. And doing some rapid cycle evaluations trying to figure out which ones seem to be paying off.

I appreciate that comment. And Mary's. One response, the struggle as the director is exactly that the resource is a pretty small. And the kind of danger of letting 1000 flowers bloom, [ Indiscernible-multiple speakers ]

Normally just that one hundred bloom. I'm just saying that you do this all the time, you can target the thing, have I don't know, one quarter of it for ideas that do not fit into what you are doing. But encourage people to take a perspective. My own sense is you always apply these things in a population or some populations your butt married to think broadly. The real issue on medication ordering or medication adherence or diagnostic test used, and you look at it in a couple different diseases but focus on the larger questions, otherwise we say this, this and this and it is harder to put it together.

David?

This is somewhat connected to this, better measures and tools are needed that relates to long-term, acute care and options such as care at home or other types of nursing home care. So for the context, I serve on our joint venture with Select, and this comment is not meant to be at all critical of Select. I have been on the joint venture board for about six years. The first years on the board I would be seeing these measures of percent of consultations turned into admissions. I do not want to see that measure anymore. I want to know what we are actually doing to improve health outcomes.

And then about two and half or three years ago when we had a mini retreat, I said tell me, what is your value added proposition in the accountable care world. And they said we don't know if you want to be first to market that space. I said that is not the right answer. What I see as a board member on the joint venture is we are not seeing the information around why was it a better idea to care for a patient in the long-term acute care hospitals as opposed to discharge the patient home or some other care setting after a total knee or total hip, and you could extend that much more broadly.

I think there is a great need for better evidence-based measures of outcomes in that space, and tools to help identify what is likely to be the best care setting for a particular type of patient. And I realize that is a huge area of research, and I am not an expert in this space at all, so there's probably more evidence in this area that I recognize that certainly is a board member I don't really see that.

And I think with the recent proposal from CMS on bundling of payments and 75 MSAs the need for what you just said is even greater.

So recently the [ Indiscernible-name unknown ] bill passed that changed the world for physician bill payment system. And one of the proposal requirements and the bill was to develop measures for non-patients facing specialties. And those are typically pathologists or radiologist. They do not have a lot of measures. But when talking about diagnostic accuracy, they come into play in a big way.

And I think it would be helpful for this funding opportunity to look for evidence, or how to collect that client data, from these specialties that are nonpatient facing but contributes a lot to dynastic errors. That is one thing. In our current scope of work as a QIO, we are working hard to [ Indiscernible-muffled speaker ] events in collecting data in that regard, especially in the adulatory care setting, is really, really difficult.

We know that the HIT/EHR can help but I do not think it is very at. - But I do not think it is there yet. And the culture of not reporting these kinds of errors is not there yet and very hard to overcome, and any kind of patient safety work I think, and in particular adverse drug events or diagnostic errors. I think that should be taken into consideration. The last thing I want to add is that the use of technologies in the home, especially as you mentioned, or even in the hospital setting watching the patients in terms of the falls, or other safety issues, might be investigated I think. There should be more evidence on how that is used and it could help save costs if they could be used effectively.

I just want to build a little bit on what Mary said related to HIT, I do see some synergies especially in the adulatory care setting and the work we need to do with health information technology it is a little bit scary to me that the computer knows which of my diabetic patients will develop end-stage renal disease, and knows which of my patients have high blood sugar the design diabetes on its problem list, it is which of my patients have undiagnosed hypertension, there is a lot of data so investing in predictive analytics and clinical decision support that you're going to be doing with this PCOR initiative would be helpful as well. And amatory setting there is a lot of work and tools that we need, it is almost there, the data is there, it is really scary. The computer knows which of my patients is overdue for mammograms, it knows so much but it is not accessible to me and the point of Gary the panel management work we are doing. Getting the last it out of there would be a great return on investment that that would be a targeted place to go, data aggregation and productive analytics and how to bring the data into life in a meaningful way that can again save lives, and certainly improve the management of patients with chronic conditions and comorbid conditions as well.

Sherry?

So I do not disagree with anything that has been said, especially about the clinical decision support, but I think we also have to recognize that how we improve the care of patients and families is also dependent on the patients and families. There is an anonymous amount of information that comes from our EHRs and what information we pay attention to and how that is used to make decisions is still, to a large extent hard.

And when you think about diagnostic errors and overuse and how you use test information, I mean it is an assumption it will end up in good. So I think there is opportunity to build out on what you mentioned, which is shared decision-making, a good example the lung cancer screening. Covered with a stipulation that share decision-making tool is to be identified and used, I think there is opportunity to better understand how we use information in the management of patients that event should translate into better outcomes, but may not.

You know, some of the discussions at Academy Health on care started to touch on that but if you take it up one level or a couple of levels it is also something that you can look at within the context of the health system project that you were mentioning. And building on David's point about decisions where people go to receive care, postacute care, it has been largely based on bed availability or what partnerships you have, what business arrangement.

We will have information that is standardized across postacute care settings through the Impact Act and other emerging opportunities. Again, how you use information that is available to make those decisions that then improve the management of patients and result in better outcomes as opposed to worse. I think that is kind of a theme I have heard pulled from different statements that were made.

So in giving you some focused areas.

[ laughter ]

With a limited amount of money.

Right. So I would second Beth's suggestion about diagnostic error in an intersection with HIT because a lot of times are attempts to use HIT to improve things often make things worse. Think of all the drug/drug alerts we get that make it less likely that we pay attention to the one that is actually important, the one out of 50 that might be important. It on the safety thing, the reason safety resonates with the public is because they can identify with that case of a patient who went to the hospital and ended up dead because of the health associated infection. - An example that you're already doing some work in is in opioid use. And obviously for the VA that is a huge issue here at Koepp prescribing, - Koepp prescribing - co-prescribing and given the rate of suicides going out and having that face it you can trace back to practices. That seems to be a more manageable kind of area that obviously there would be a great chance to partner with us. If you wanted to do something like that we would work to see how we might cofound some things. It is also not a simple challenge because changing people's opioid dosage is not something that patients necessarily jump to. Trying to come up with better alternatives for pain treatment is a challenge. But that is something where the outcomes are pretty start and identifiable and the rate of adverse events, short of suicide, is relatively high that you actually might be able to show a benefit.

Thank you for solving my problems.

[ laughter ]

No, it has been very useful. Both in affirming what I have heard that the fairly open-ended approach we have taken to date probably I have heard a lot of support for that, but as well, a lot of good suggestions for more focused approaches with may be pieces of the investment. Again, I would be delighted if we could make a much larger investment and do much more to focus on stuff but that is where we are. You know, I had a second topic which I am going to just introduce and ask if anyone has any thoughts about it and to just send an email.

Very unlike what we just discussed. We said in the 15 budget that we were making a $15 million investment in extending patient safety to all settings. This second thing is just kind of a question of should we do anything? We all know there is a tremendous amount of interest and buzz around healthcare organizations, IOM has done a lot of work on this and many others, and as I discussed earlier, we are making an investment into the three centers of excellence, studying health system performance and PCOR dissemination.

My question is we think learning healthcare organizations are totally a great idea. That what is a learning healthcare organization and would there be any value in trying to actually develop a measure of whether a healthcare organization is a learning organization? And then what the relationship is of the different parts of that to the outcomes that we all want?

[ Indiscernible-low volume ]

[ laughter ]

It is at least a question. And that really is the question. This is a graphic that Mike who is not here today put together based on some work and also Sarah Singer and also [ Indiscernible-name unknown ] with Kaiser, what are the components of a healthcare organization and her other questions. But we are at a break.

[ Indiscernible-speaker too distant from microphone ]

I think this is really great and timing it with what you've said earlier. [ Indiscernible-muffled speaker ] [ laughter ] and the family has gone back to where they belong. [ laughter ]

You know, for me the idea of identifying parts that can be studied and acted on. Everybody knows healthcare organizations, all organizations have to learn and evolve and respond, but I think we all struggle with how and where and why and what makes a difference? So bringing some order to this chaos would be extremely useful but keeping in mind what you were emphasizing earlier in your remarks about the key role of the agency and the mission of the agency to inform people in a way they could act on that information to improve health of the patients in the health of the public.

Thank you. Again, any other thoughts about this, I think you all know where to find me. I guess the last of the agenda for today, we're at the break and will have discussion about the QDR and the public comment period and will have a speaker with the Surgeon General's office to talk and speak but the Surgeon General's priorities. And in the afternoon will talk at work we are doing and look very much for your advice on what we should be doing around transparency issues.

Another agency does not have money for watering and feeding, but if we could get some napkins, you know, because some of it is like strawberries. It is all been washed. My hotel served us a great one last night.

[ laughter ]

Many thanks.

Will take a 15 minute break.

[ Group dismissed for a 15 minute break. Captioner standing by. ]

Thank you Beth. So to words or two sentences, equality and disparities reports are done annually by the agency and Ernie Moy who is an interest has been involved with it since the very beginning and has been in charge of it for much of that time. They are I think the quite valuable resource, although one I have been struggling a bit the last couple of years I have been here to try to figure out how to make either more actionable, which I am not sure I know how to do, or certainly to make higher visibility and have more attention paid to them. Which I think we ought to be able to maybe do. There has been a lot of interest in this area, the Institute of Medicine, David Blumenthal shared and issued a report probably six weeks or two months ago talking about the need and was parsimonious and well understood what we are doing and a lot of that is what the QDR is. I have asked Ernest Moy who I said is and internist, to provide an overview of the reports and kick off a discussion of your suggestions and advice about the substantial changes in the last year.

Thank you. I will start by saying we are totally open to comments, when the doctor came on he said do the reports and this is what he came up with so I can always blame him. [ laughter ]

That this is an experiment in some ways it is quite different from what we've seen in the past and want to get your input on what is progressing, what is working and what is not working, as well as how we ought to continue to change in the future. I will focus mostly on the changes in the reports but I feel like I cannot do that without giving at least a 32nd background. These are reports that come from authorizing legislation back in 1999 and we've been doing them every year. This is providing an overview of the quality of healthcare for the general US population and disparities in care, focusing on racial, ethnic and socioeconomic groups that extending to all publishers which you probably know the long list and encompasses almost everybody, unless you are high income, white male maybe you are excluded, but maybe everybody else is included here it

We see the purposes of the report to assess the performance of the healthcare system globally and also try to identify areas of strengths and weaknesses. Kind of a report card and targeting. Targeting where we need to meet more effort. And the two basic dimensions of the report are access to healthcare and quality of healthcare. Healthcare is a common word. This is a healthcare report that we were tasked to do its agents for healthcare quality, wherever we are, research and quality. So again, healthcare is operative word.

And is submitted on the half of the Secretary. of Health and Human Services . I want to spend a couple minutes on measures and data as I think people are asking us frequently about where we get the measures from. First of all it is meant to be very broad-based and so the perspective of the report is we want to show how the whole system, the whole healthcare system is doing as a consequently we try to cover many settings, many aspects of care and clinical conditions, we've come up with about 250 measures.

The way we can do these measures was way back when we started there was an initial set developed and this was developed using input from several Federal Register polls or calls for measures from the public. The national committee on vital and health statistics and the Institute of medicine provided input and there was a method workgroup in the department that took all of the suggestions and distilled it down to the initial set of measures retract. Every year we add new measures and we get suggestions that come in from other US agencies and other suggestions from stakeholders and all of these are reviewed by our interagency workgroup which is HHS, almost all the operating divisions, and they approve the changes to the data and measures the comment are reports.

One of the issues there have been questions about whether or not we should change this process and try to get other clients and stakeholder input and we are certainly hoping to or open to it. I just wanted to describe historical process. We are constantly changing. I want to talk about some changes today, but every couple of years we make pretty significant changes to the quality disparities report. This is our timeline from authorization in 1999 to the first reports in 2003. And you'll see additions basically each year over time.

We went through a very significant reorganization back in 2010 when we tried to bring to reports close together by having a consolidated highlight. And this again is another major step forward I think. We have a fully integrated report and we have integration with the national quality strategy priorities and chart books and I will go through each of those aspects in a minute. So the report. How they have changed. First of all integrate findings from both healthcare quality and disparities into a single document.

We thought this was important. Our intent was always to say the quality and disparities are both important and both need to be viewed at the same time if we really want understand the healthcare system. But we did have two reports and I will have one. So that forces people to see the full integration of it. If they only want to see half of it they can't, they have to see the whole. Focusing on it summarized information, which we always have, organizer on access to healthcare and quality healthcare. And this is organized along the six national quality strategy priorities. They are way, way smaller.

I brought along last year's reports, they are 200 to 300 page document. And this was 30 pages. I think that has made a big difference I can tell you about that in a second. We also really sped up the production of these reports, and part because wanted to get some ACA information into its, but overall I think it has been a general improvement. In the past we would only take data through the spring of the year and they would have about nine to work with that data to produce a report here we are now taking data up to the fall of the year. We are taking data from September and October of last year and we took special insurance data up through December up through December 2014. We got these out right away then.

There were some challenges we were asking our partners and the department to clear things without the final numbers and without that last data point. We were surprised and actually encouraged that they were willing to do that to get the most recent data out there. Very quickly, these are the national quality strategy priorities, we still have the three aims and we have six priorities, you probably are all familiar with, and we have the nine levers. This is the national quality strategy eyeball as we call it.

[ laughter ]

And I think the key findings of the reports, however, will not surprise you totally. We have said many of these things in the past, with the exception of our statements about national quality strategy priorities. On this report it found access improved, insurance numbers were dramatically better, we found that quality improved which is a statement we made in the past but now we're able to track at least a couple the national quality strategy priorities and we are headed in the right direction. In contrast disparities are still a problem found a few spirit is were eliminated and we try to attend by the specific areas were quality and disparities seem to be having the most difficulty having or making progress.

And I will say very briefly before I turn to the chart books, this new and very short report has garnered us some benefits already. People have come to us in the past two of said that was a nice report and we cannot say if they thought it was nice because I had a nice cover or if it was nice is a doorstop. But those people now come to us and say oh, I found this particular figure very interesting and they would ask us questions that they had actually open the report and look at something. It is sad to say that some people can guess and said that we found it was interesting that quality and disparities do not track, but the biggest disparities or not found in the poorest performing states for I was sad to say yeah, and this is the eighth year we have shown that. But I do think the people are using it more and their opening it and ask questions.

So would take the two big documents and make it into the tiny document that might be stakeholders out there who are mad at us because they lost all this information. We knew we had that information out there in some way your we have been doing is putting them out as a series of chart books. This is where all the information are historically included in the reports on individual measures are found, they are still in both quality and disparities and organized around access to healthcare and the national priorities and the way the series coming out that is the priority populations. These are electronic only.

They are available on our website. They are available in multiple different formats, there is a PDF and PowerPoint format such as take the slides and use them. We have been releasing them every two weeks for the last couple of months. And that is actually given us a number of benefits as well which I will show you. The website itself, the supporting data are all found on our NHDRnet website that we are continuing to develop, where adding navigation that will track the national quality strategy priorities and also tracking it using the digital analytics program, which is the Google analytics program for federal websites.

And we have been busy. Chart books covering these topical areas, there were two that came out on April, two came out in May and two came out in June and it is too small for me to read. I know have covered access to care and all the national quality strategy priorities except for the effective treatment of [ Indiscernible ] and that will come out next week, now we will turn our attention to the quality chart books. I want to taste a quick look at what you can see in these chart books, I will tell you about it. This is the PowerPoint version and what you will see in the PowerPoint version is there will be a PowerPoint slide and in the notes section you will see all of the supporting detail.

You'll see why it is important as well as the interpretation of change over time in any disparity information shown and so on if you pull this in the notes page which is how we want people to use it, you'll see the slide on top in the description of information at the bottom. But then there is also the slides if you want to import it into something. This chart book adds a lot of flexibility. You cannot see it in this way. One of the things that we were able to do is add new kinds of media. I don't know if I can actually get this to go. This is the Surgeon General's conversation with Elmo about childhood ministrations. So that is embedded where we talk about childhood immunizations. You see the video and then you see the data that we have on childhood immunizations, trying to make it more interesting to people and give them multiple things they can use if they are making a presentation.

And this is where we are, we had our last national quality strategy chart book coming out next week on effective treatment and they will move to the priority population chart books. And this is I think where we are making the biggest, I think, potential dissemination. I do not know if you remember the last report I passed reports that priority populations are the last chapter of the disparities report and each population had basically about four pages. Needless to say all of our stakeholders were upset that you cannot say anything about their population within four pages.

Within these chart books each party population will get its own chart book eventually, we are doing them over time and we are putting out three this year, although we started off to put out four. And these I think will be much more useful for advocates for these different populations and we are working with the people who work on these populations already. The one that will come out in August is on rural residents and we're working with the office of rural health policy, over at HRSA, NCI, and ASPE. And I think they have a use in mind so I think it will be more useful. I think the chart book also at a lot of flexibility. We're letting these folks tell us what is most important. Overall residents they said patient safety is not like that big a deal in rural areas so we're having a small section on patient safety though it is still included. Access to care is a big thing, we have a quarter of the chart book for access to care. I think this format allows a great deal of flexibility. I think it will really be directed towards the needs of potential users.

And you can see that we are working on the women chart book next, one for Hispanics was requested from the national Hispanic health alliance and we are also working with OGA on that particular chart book. Wanted to do one with multiple chronic conditions and as we were chugging along we realize we had to redo some of our analyses to we're putting that one off until next year but we were putting it out then.

Justice - this is really quick and I do want to emphasize the tiny thing which is the NQS priorities thing on the left and I just wanted to show you that. This is related to patient safety or person centered care and it will give you that bundle and show you how that bundle of measures is doing. These are some of our web statistics. Can you read it? The left arrow pointing down is last year's release. The second arrow is the release of the state snapshots last year. The third arrow is this year's release. I think we're seeing a bump up, at least in the web statistics for this release, and we have not promoted the release of the state snapshots yet. This is historical web statistics.

This is the digital analytics we have added this year which allows us to look at it on a real-time basis and summarize on a daily basis. This Vikings - the spiking have been the web traffic since the release, sometimes within more sustainable have historically been we put the charts out every two weeks and so that is why we see these spikes. Attention is drawn to it then. We see double spikes, we see an increase on Wednesdays even when we don't do a release. I think in some ways we are training our potential users so they know to come in on a Wednesday to look for something.

Also do a lot of stuff related to tracking specific kinds of rollout activities. We can see if a particular chat or webinar is drawing attention to us. We contract good and bad landing pages and we can see where people are coming from, it is not just for Maryland.

What is a bad landing page?

It is one that bounces. You can track bouncing where people come your page and I leave right away. That is a bad landing page.

[ laughter ]

And you can see our statistics compared HHS overall. People are spending on average of three minutes on our site, an average of 1.7 minutes on HHS. We're seeing 3.2 pages on our site and 1.6 pages on HHS and we bounce about 44% of the time or is HHS ounces 74%. These are our immediate plans for the next cycle because we have to start gearing up for 2015. And again it is proposed by folks from the CDC and I felt will have one measure on smoking cessation which was an adequate and they wanted us to add measures release on medication use or use of medication to quit, the past years quit attempt in recent smoking cessation. We're adding one on breast-feeding. Now is the questions part.

First of all what do you think but also how do you think we can make this better? This is one of the ways we review of the quality and disparities report. Now that we have this whole integration with the national quality strategy, we think that it tells us what is important and we track the things they tell us are important and we try to direct people to the implementation resources available at AHRQ as well as outside the department so that they can make stuff better. I think we been successful in drawing people into the reports themselves so these are researchers and educators who want the information, that we have been less successful with driving stakeholders to the implementation resources and so we seek your guidance on how we could do that better, and number two, reading it back up to the policymakers so they can improve their plan on what we ought to be tracking.

Another view of the quality disparities report -

Ernie, order the two areas that you think your weaker?

I think were doing okay reaching the stakeholders, but I think that we are doing less on this, we want people to come and see a problem in essence and then say what to do about next? And I think that can be found at AHRQ, we have tools to help people make those improvements to trying to make that direction or directing them to those tools have been one of our problem areas. And the other thing is when he defeated back to the policymakers about the successes and lack of successes in the current plans and have them rethink some of their policies perhaps. I think we have been less successful with that feedback loop as well.

To expand on that, this is an analogy that does not work, but that is part of the problem. If you think about economic statistics, unemployment comes out or GDP or whatever, those numbers used by folks who are making decisions, both in the private and public sector. These numbers come out and they do not much get used by folks making decisions and I think when Ernie is talking about lack of feedback, maybe they can't be or won't be.

I was wondering if you guys do any kind of targeted pushing. For example, health reporters, on the web and in print, particularly New York Times, Washington Post, LA Times, Chicago Tribune, Boston Herald, that the other folks read, send an email, I have used it though I've stumbled across it of it. Every time I talk or when I teach I would use it in my courses I would say 20% of my slides come from the Kaiser family or Kaiser foundation, which is also easy to use and has good information and sometimes it is from you to them. I was thinking are there key policymakers? Talk to David or people like David and find out who in his organization, who are the people in the organizations who would find this useful? And then send them a link when the report comes out and stuff. But not necessarily send them every you link. I think all of us find it hard when we get six things from AHRQ and some things are more interesting than others is what I tend to do as I say I will come back to them later and usually I do but sometimes I don't. I think thinking about key audiences you want to reach in the private sector or public sector and making sure they know about this, and also with whatever their organizations are to you have the key staff linkage they can do the pushing for you.

In terms of action I think one of the things you might encourage and even help your consultation would be to encourage health systems to replicate this analysis and apply the logic to their own populations. If you look an easy this is the whole country but I do not take care of the whole country and my system is fine. It wasn't until we did the racial/ethnic stratification of our HIDIS results that we started to expressly reduce the disparities.

David.

One lever would be the various profiling efforts related to the performance of healthcare systems, particularly in the private sector. So you have these various groups like Thomson Reuters who does the ranking of healthcare systems, and there may be an equity dimension to those measures, I am not sure there is, but in the world I live in, we benchmark against 17 other or 18 other multi-hospital healthcare systems to determine our long-term incentive program, which account for probably 40% of the compensation of senior-level executives in our organization.

And mainly by hospital compare performance, Molly Joshey does this analysis at DHA, if yet the opportunity to incorporate into those analyses dimensions of disparity were able to characterize a given healthcare systems opportunities and disparities based that could be quite powerful to advance this work.

As you know QIOs are a major consumer of AHRQs tools and resources and one of them is this. In the past you had presented to the QIO community is but I have not seen that lately. I can certainly relate the message back to our QIO community but it would be nice if we could get a webinar done to announce this, especially the new change, which is really nice and format. That would be helpful.

I was looking these over on the way out and I think you have done a nice job Ernie with these and the new format is better to let me give you that feedback is a user I do like the concise one. I like combining the disparities and the quality. I think it is useful but it sends an important message that is one I am trying to do more which is not separate disparities in quality, it is sort of a core measurement and all healthcare organizations should be thinking of disparities, not as something over on this side, but when you are looking at quality break it down to the subgroups. I think it is that message and you may even want to say or articulated, there is a philosophy behind that, right? And I don't know if you fully state that. My guess is you would agree with that.

Oh yes, absolutely. I think it is more than you just thought it was good to have one short book over two, it is just this is our philosophy, disparities and equity as a core part of quality, it is the IOM quality frameworks he may want to emphasize that because I really support that I think it is great that you did it. There is something lovely about things that are concise. I can those tables summarize, quality indicators that are and how many are getting better, worse, disparities that are better or worse. The only thing with the next round, it would be able to click on that and then see what the measures are or something. You could get your 2.01. I want to know what those 19 indicators are for access and things like that and then probe those of what - probe those a little bit deeper. Your NHDRnet, you can query things?

You can click on figures and plot the measures that we do not have all the graphics represented so we're focusing on relative benchmark that we could try to put in all the graphics that you see in the report.

Yeah. I am not sure from the chart but, you noted there is a data query look up?

I have to figure that out a little bit.

For people like me, the simpler, when you are on a chart book that it is clear, you almost want on every page, there is a link if you want to get the answer for your own state or if you want to get the actual data and things like that. Because it is a little bit parked on that site that is a little bit separate in the short book. Those are minor things but I think it is a really good what you've done with it.

How well can you compare or do comparisons on the website? If I wanted to compare one state to another could I do that simultaneously? The reason why I bring that up I am in Georgia with a non-Medicaid expansion state, urban articles that are come out the last couple days comparing Cook County to Grady and who is doing well. I'm a policy perspective it would be helpful to me to go back to my governor saying here's how we compare to another state expanding Medicare and I can put directly off that website in an interactive fashion.

The structure of it is not like that. Look at one state at a time. That what we do is when you drill down you can see how all 50 states do compared to you. But we could make it have a left and right where you could pull up one state and have another state on another panel, that would be conceivable I think.

It is nice, you can look up George and you can separately look up Illinois.

I have done that but it would be nice to do it side-by-side. If you pull up George agency all other 50 states if you find Illinois that it is a hassle.

Ernie, along the same lines with pointing people to implementation resources, how obvious is that off the chart books? I haven't had a chance to play with them online.

Right. It is like a lot of our linkages which is probably not obvious.

[ laughter ]

You have to look for them and we have been linking over to things like the healthcare innovations exchange and NQS priorities in action but it is probably an area that we ought to figure out how to expand more is it felt like that would be helpful.

And one other possibility, I would imagine some sort of companion or case studies where you could say how people have used our reports so that you could point, well, it may not occur to people what is in your head about how you could use them, but you could have some little demos of creative ways in which people have leveraged information in the report was some kind of guy about how to do that. And really one or two things good light a bunch of people on fire, including pointing reporters to it or having a slow day, it could be a great place for them to go to highlight something related to the area, the local area they are writing about. Which in turns would generate more traffic.

Along that line, using this report to not drive people to other parts of your website where the tools and resources are and where evidence of best practices are in translating practices into improving care, I think it could embed in areas where you say this area is not improving, but here are some resources that you can use to improve this particular topic or whatever it is.

Sandy.

One of the things I have noted with reporters, there's been a general cut back his health in particular. There has been a small, but not insubstantial group of reporters that are doing investigative reporting, that most of us would call research, like Noah Leavy had a great article in the LA Times the could easily have been a research study. You know there are a handful of key people, no at the LA Times, [ Indiscernible-multiple speakers ] and Rob when the and his group of the Wall Street Journal, a mentioned by them or incorporation into one of their stories, everything that everybody else of the room has done on steroids. Our steroids legal for that? I don't know.

[ laughter ]

I just think those people in particular, more than the TV people as they are doing soundbites or stuff like that, but on the upshot, one of these blocks are one of these key reporters, they're really doing substantive work now and they're getting more into the weeds themselves with the data. And I think that you could easily have this disseminated very quickly through them.

And I think the other -

It is useful to them.

Yes and with some of those I am thinking of the upshot of particular, like the health section in the New York Times and Wall Street Journal, their embedded hyperlinks could drive folks to your site.

Exactly.

It is still amazing. Definitely an opportunity. I think they would very much be simpatico.

And if your communications people would ask for who they personally know, as Mary knows, she will get an email from a saying Sandy Schwartz and contact me.

[ laughter ]

She knows it is okay. And I think that you can say I am contacting you or you could write something for one of us to send to them to establish relationship.

You are hoping that is Mary's reaction.

[ laughter ]

Now. I said Mary gets it but I don't know what she will do with it.

[ laughter ]

She has a filter for that. David.

It seems like one place that you could learn the most or there is the must interest is in variation. Striking variations state-by-state or variations over time. I think snapshots by themselves don't mean anything. The question for policymaker is which of these can actually move? I do not know if there is a possibility to highlight some states that have made substantial improvements in measures and speculate on why, or, to Leon's point, the difference between Medicaid expansion states and non-Medicaid expansion state. Obviously you're getting into more speculative things, which I don't know how much is really in your purview, but it is those unexpected variations that I think will catch people's attention.

Because people are used to seeing the South doing worse, there used to seeing this different measure, and so a policymaker sort of often feels like yeah, I am always 49th. And everything should be a priority. But if they could see a statement looks like them but is done differently that raises the question that oh, maybe there is something. Or if you contract states that maybe you have known they have done an initiative around a certain project, it is important to figure out who audience is to the extent that most of this is state-level data, the audience are those who on the state level stuff and I know that AHRQ has partnerships with the Medicaid directors and the other people who are state-level policy makers.

Have you done anything focus group to understand what they're looking for?

We do webinar for the state Medicaid directors every year. We do make improvements based upon their suggestions. The state stories, I will say it is hard and we do try to do that. We find states that are improving quickly and we search their websites. Every state says it is doing something innovative and healthcare right now. So we do not want to go out and say this state is doing something innovative and they improve quickly but other states say they are doing so and then they are not.

And so the IOM core metrics, I am wondering if you're planning, there you are.

This is the other big question that we have, were doing okay and then all of a sudden have this. Questions on how to interact and deal with this because we cover roughly half of the vital signs. If we covered a little bit we would say ignore it and if it was a lot we can say we cover the whole thing but it is literally about half. What we do with that? Do we try to cover the other parts? It is not in our chart or not and or scope or given to us in our legislation. Do we try to find partners? They can help the people and bring the other half or do we find someone else? That is just one of the questions and we are not sure what to do when somebody says that you are doing a half job.

[ laughter ]

Vital signs, trying to come up with a parsimonious set of measures, there are 15 domains and in each domain there are multiple measures so it is not clear how parsimonious it ends up being. That is pretty different than the QTR which has a 50 measures although can be summarized into variety of domains so not so different.

I make two observations and one is that the word measures or metrics in that report is used rather loosely as a member of the committee I can say that. So there are not measures for all of those things so do not feel like you miss the boat. The other thing is I think there is a fairly big hand off in the report to the Secretary to come up with a report and if you thought that you had a play there I think it could become part of your purview by saying to the Secretary that we can pull together, to the extent, because some of those measures to sit in other places and it might just be a matter of developing a report that picks up on that because it is not clear who else would do it, you know what I'm saying? That might be an opportunity.

Thoughts?

[ Silence ]

We are always open to suggestions. Since stuff over to us and as you can see we have the ability to adapt significantly.

[ laughter ]

I just want to congratulate you because I think this is a really nice step forward. I think it is definitely in the right direction. And I think the bite sized chunk they are rolling out I'm also from the perspective of getting attention in the media, is definitely helpful. And that is really big is much harder to cover. So watch this space or coming next week is really terrific. And the usability, I have seen this and I think this is maybe the Kaiser family foundation who was the early adopter, but creating things that people can dump into their own slides is brilliant in terms of dissemination just because people will not go through the trouble of making PowerPoint so if you have an easy access way, then I think that will really get the word out. My guess is you have not seen the half of it yet. And you'll probably also get more suggestions as a result for things that people wish they had as they go to try to use it.

I just wanted to ask, and the communication pieces that go out about the release of the chart, does that include a section that these can be downloaded for use in presentations? I just want to reinforce that the usability of those charts and the PowerPoint is extremely valuable for people who have been making presentations. You just made half of their slide set. I think that should be included in the communication.

We do not as a that is a good idea.

I do want to leave this meeting was something very practical, downloadable slides, they are available now?

Yeah.

In the chart book? Would you flip onto the chart book and show us?

[ laughter ]

I am going to represent the really incompetent people coming here. You go to the report like the chart book indirectly download from the page?

You cannot download from the page inside. When you first come to chart book there are multiple versions available to you. You get the PDF version or the PowerPoint version.

I see.

At that site you can pick what you want.

When you first go.

It is the Do My Talk For Me.

Or in Sandy's case do the course.

[ laughter ]

I find it very useful because I don't to make that many slides so it's great.

As I understand it we do not have anyone registered for public comment. So what we are going to do, because we just have Dr. Burmal for 30 minutes, from 12 PM to 1230 12 PM to 12:30 PM, she will be with Stan and whisked out there it we will suggest the people take a break and grabbed their lunch and be back and quiet and attentive in their seats at noon. Thanks Ernie.

[ Group dismissed for 20 minute lunch break. Captioner standing by. ]

[ Captioners Transitioning. Please stand by. ]

[ Captioner on standby ]

[ The meeting is on a 20 minute recess. The session will reconvene at 1:00 EDT. Captioner on standby. ] We are very happy to have 1017 here who is the Director of Science and Policy, Office of the Surgeon General, Office of the Assistant Secretary for Health U.S. Department of Health and Human Services. We just discovered there are common roots and that she was on the faculty at UCLA and did training at UCLA before coming to the Surgeon General. She has been practicing primary care, is a clinician and Carter Andy from Harvard Medical School and also an MPP from JFK school and health management from UCLA. That -- she is well-qualified which is the hallmark of the Surgeon General's office these days. We are excited to have you here and to hear what is up with the Surgeon General's office.

Thank you so much. The Surgeon General sends his regards as well. AHRQ is a special home for me because I joined the Surgeon General's office in March I joined the Surgeon General's office in March 2015 to help Doctor Mercy to [ Indiscernible - Intermittent Audio ] Are is a very special home for me because -- I joined the Surgeon General's office in March -- I joined the Surgeon General's office in March 2015 to help Doctor [ Indiscernible ] to move forward his vision for the health of America.. Prior to that my professional identity was one with a focus on health disparities, primary care innovation and chronic disease prevention. I continue to see patients here in Maryland. It is very post to my heart and I love the work and have been touched by the work. I know our time is short so I will go through the priorities of our office and would welcome your thoughts on synergies. It is a challenging time for art I think even though her -- millions of Americans have been covered by the Affordable Care Act we in this group of specialists -- very accomplished experts know that coverage is only one aspect to better help. -- Help. There is so much more to do. We have high rates of obesity. One in two adults has a chronic disease common diabetes, cancer, heart disease. Half 1 million deaths continue to happen because of tobacco-related illness. 42 million Americans still struggle with mental illness. We spend an incredible amount of time investment -- investing in treatment. That is great because of all of the advances that we have made in treatment. This helps the patients that suffer from illness. At the same time as a physician myself, who has had the benefit of delivering treatment, it can also be very frustrating for patients at IC who are diabetic who come to the hospital for treatment every month for cellulitis. Or for the person with COPD who comes in for exacerbation because of a lifetime of smoking. It is frustrating for me to cite, what could we have done before they had come to our doors to prevent diabetes from happening in the first place or the COPD or tobacco-related illness. The truth is we really don't spend as much time at prevention. We are not as good at preventing illness as we are at treating it and that really needs to change. I think this group knows better than any other group that having a more expensive healthcare system does not necessarily translate into better health the US still ranks low or last in a number of indicators such as and for mortality, quality-of-life, efficiency of the healthcare system. While these trends are concerning, I think we can overcome this, and our office wants to overcome this trajectory through a culture of what we are trying to build, a culture of prevention. That is rooted in two things, this idea of community prevention and health equity that runs the gamut and is so dear to my heart as I see patients often in underserved communities. Creating a culture of prevention is really about what we want to do in our office. It has two strategies. It's not just about the programs and policies that you put into place. Let me be clear. The Surgeon General's office does not make laws. Is not a regulatory group. We don't enforce. We have an incredible ability to communicate. That is the second part of what we want to do. We don't want to just think about the policies and programs in place to create the culture of prevention but to go a step further, which is to change the norms, beliefs and attitudes and behaviors. In terms of creating this culture of prevention, we have two roles at the Surgeon General's office. One is the communicating of important health information to the nation. The second, which I think is a lot less known but is equally important -- the reason that Surgeon General Murthy wears uniform is to help with the people in the mission court was mission is to protect and advance the health and safety of the nation. A are wonderful force to help move this vision forward. When we are thinking about a culture of prevention, it is not a structural change. The cultural change is a lot to do with a message for what our office would like to do, one of the main priorities would be about how we communicate that information. -- Modernizing how we can you -- communicate that information. People don't necessarily receive information by reading a newspaper cover to cover. I know I still do but I will tell you what I have learned as a scientist, coming into a health communications bully pulpit role, that's not the way the majority of Americans receive information. It is not through CNN either or these wonderful iconic comprehensive Surgeon General reports that don't necessarily reach the public or are not necessarily actionable, nor through speeches to an audience. When we think about that -- I heard in his previous talk a clip of the Surgeon General with Elmo. When we are thinking up message of prevention, to make the healthy choice the easy choice, also to go the extra step which is to make healthy choice the desirable choice. It's not just about the mode, it's also about the messengers and how we partner. When the measles outbreak came upon the center will -- came out, the Surgeon General came out with Elmo to talk about vaccinations. One of the things that we found with that mode of communication was the importance of the messenger. Elmo is trusted and so is Sesame Street among kids but also quite frankly among adults. That video that we did, if you haven't had a chance to see it, you should. It is funny but it is actually really an I -- it also has an incredible amount of health information about vaccinations. I was surprised. It's not just a funny skit about this -- the fact that vaccines don't hurt. It's about vaccinations, antibody somehow it protects and why people should or should not get vaccinations. When we -- what we heard when it went viral was a kids liked it but what really happened was parents appreciated it they appreciated it and saw it. The feedback was that they felt like they understood vaccinations but sometimes when you are on the fence -- as a mother of a three-year-old and five-year-old I can attest to this even though I am a physician. When you see your child getting five vaccinations in one visit, you do stop and hesitate and say, are all of these needed right now. That really helped allay a lot of fears. They showed it to the kids and the kids were excited to go to the doctor instead of having fear associated with the doctor in the white vote -- -- white hope -- coat, or really the nurse. The messenger is not always a celebrity. The messenger is often people at the local level, the people outside of the health sector, the teachers, our schools, the YMCAs or civic organizations, faith-based organizations. These are all groups of the Surgeon General's office wants to collaborate with as we know forward this vision of a culture of prevention, rooted in community-based prevention and health equity talking about structural change come out the healthy and easy choice it also the cultural change, which is the health of -- healthy choice is the desirable choice.

How we're going to do this is we are going to execute campaigns in these areas. I am going to talk to about the five areas where we are going to spend a lot of Doctor Murthy's four-year tenure for when he is confirmed -- it is not two years. It is not related to the administration. The office is protected from not for this exact reason. Five campaigns, we are going to be working on, the priority areas. The first one is on active living. We know that half of adults don't get the recommended exercise, physical activity that they need. It's even worse for kids. Only one quarter of kids get the 60 minutes that they need every day. We are putting out later this year, hopefully in the fall, I call to action on walking and walkable communities. We have named it, "Step it up." Is not just the idea that walking is a physical activity. It is thinking about walking as a place or social connectedness, a place for emotional well-being, a place to think about getting snacks of exercise or fuel for your body, not just the health message. Even more important and that is the idea of walkable spaces. The truth is a lot of people don't have access to that. I can attest to this myself. I often prescribed exercise as medicine I start with walking. I have to say 99% of them are obese. Most of the patients come back to me -- I remember one specific one -- woman who was a young mother. She said I can go walking because I am scared of getting shot and mugged in my neighborhood. That is a reality of what we are talking about today. There is this great example in Indian River County Florida were 50% of the population lives in poverty. In 2004, the Indian River County health Department wanted to do something. They notice of people -- their health conditions were getting worse and it was attributable to poverty. They took it upon themselves to make improvements in the public face, to change the environment. They did a survey two years later and what they found was that 95% of residents said that they exercised outside much more than they did two years ago. What was fantastic is a attributed it to the sidewalks into the safe places to exercise and they said that improve their quality of life. We see these examples happening at the local level. It can be done. That is one of the things that we want to highlight with a call to action. Walkable spaces are necessary aspect for physical activity. That is one of our main priorities, active living.

The second priority area is around being drug-free and tobacco free. Here we are going to focus on the prescription opioid abuse issue that is curbing America right now. We have seen a rise in prescription opioids over the last 20 years in that is also seen an rise in unintentional overdose deaths that has tripled during that same time period. What our office wants to do is, here is whether could be some nice synergies specifically with AHRQ where we want to take a role with prescribers and providers and talk about, this would be a call to the profession and what they can do, with the good intention of wanting to treat pain, not realizing that they are actually the cause of this epidemic of prescription drug abuse. That is an area that we are going to spend a lot of time talking about. One of the stories that Doctor Murthy always talks about is -- he likes all of the Surgeon General's before him but one of his favorite ones is the iconic Doctor Coupe -- during HIV-AIDS -- the epidemic, he called upon them to do something about the HIV AIDS epidemic. Physicians took ownership of that to the point of where Judy Palfrey, someone though some of you may know she is a past president of the Association of pediatrics -- she received that letter when she was in training and a pediatrician and she got this letter from the Surgeon General and said -- is a something that I am led to do something about. She went on to think about safe sex practices, putting condoms in vending machines at my clubs and even within our own hospital Coop. One of the things that she had to do though she worked at Boston Children's Hospital -- one of the things she had to do was she decided to order 10,000 condoms from the hospital and of course procurement was red flag. Why are you ordering 10,000 condoms? She said I am doing this because I want to establish that -- safe sex practices and it is the role of physicians to do something in curbing HIV-AIDS I have this letter from the Surgeon General saying that this is what I have to do. Procurement said, okay. Here are your condoms. In a similar fashion this is the same sort of thing that we want to do with providers including not only clinicians but pharmacists, dentist, nurses, about what their role is in curbing its epidemic and to call upon the profession to take charge on this. Another area is going to be tobacco.

The Surgeon General's office has had an incredible legacy around tobacco free America. We will continue to do that especially focusing on impressions that young people get related to tobacco and new devices to deliver tobacco and nicotine. That will be a strong role in mitigating that health information, especially at a time when some of these new devices are way ahead of the science. That is priority number 3 -- priority area number 2.

Another priority number 3 is around mental health. We thought about the different roles that the Surgeon General's office can play. We feel the most effective role is addressing the unacceptable stigma around mental illness. Here we see a large -- a nice partnership with the faith-based community to do something at the community level to start some of those conversations around mental illness, to normalize it in the same way that we normalized so many other chronic diseases such as diabetes, and to address the fact that 50% of symptoms for people diagnosed with mental illness, shoulder symptoms at age 14. We sometimes call this the chronic disease of a young. This is a nice partnership in terms of the faith-based community. I think we are thinking this would be a community clinical partnership there was only so far that the community can start the community -- that the community can do. They can start the conversation but the role falls among the providers.

The two other areas are around healthy eating. We will likely focus on sugar, specifically added sugar. We will also focus on ending violence in America. One of the last areas which is not necessarily an area but a female runs through all of our campaigns is around emotional well-being. This is about a person's happiness, but for this group it is the sense of self-efficacy, the sense of purpose and meaning and baloney. Research shows that only 10% of that is infected -- belonging. Research shows that only 10% of that is external factors and 90% is internal factors. When we think about civil practices such as talking with your friends or social connectedness, gratitude, meditation, when we are thinking about making that healthy choice and desirable choice, we are really going to be focusing on our emotional well-being as a priority area. Those are our priority areas. Blessing on leave you with this some of the levers of the Surgeon General's office, I said earlier, we don't make laws. It is a small office. It has a huge omission. We don't do -- it has a huge mission. We don't do regulation. We have wonderful levers that I would like AHRQ to think about as we think about collaborating . I talked about communicating -- we talked about the five C's -- communicating in a modern way that reaches laypeople, not as policymakers and researchers. Number 2, we call it the convening role. We have a wonderful ability to convene natural and unnatural partners. This involves many groups out in the community as well as celebrities and people in music and movies, athletes and sports. Be third C that we talk about -- the third C that we talk about is [ Indiscernible ] when Doctor Murthy took office he talked about though she did a listening tour to go to communities and hear some of the problems that are happening. He wanted this to help inform the framework that we have developed in terms of the culture of prevention. What he actually found was that -- he found the solutions people were so hungry to hear what other groups were doing -- I think this group can understand this -- with healthcare institutions, understanding what X is doing and how we can relate to what are white -- how we can relate to Y and we have's ability to highlight and connect that. The fourth see is champion -- C is champion. We want to use are very effective Twitter feed on Facebook and other mechanisms to champion and highlight work that is happening all across the nation and quite frankly the world. The last C is one of the most important ones, the ability to command. He commands be 6700 US health troops and they are going to be integrated into the workforce. They are meant to be integrated into many of these priority areas and the campaign that we have talked about. You have Jeff who is a commissioned Corps officer with the US Public health service and a wonderful connection for us with AHRQ . We will continue to do that. Those are our priorities. I really wanted to sent to her some of your questions as well as thoughts and suggestions. What is a think you very much.

-- I want to say thank you very much.

Could you say more about violence? What have you been doing with that?

Some of the things that we have been thinking about in terms of violence -- this is still pretty broad -- I don't think we want to narrow it yet. One of the areas is against volumes -- violence against women. We often have this tagline, when I was in college we had these -- take back the night marches which were led by men. We often say the responsibility for ending violence against women starts with men. This would be a very limited campaign where we start a conversation around violence come on violence against women, its impact -- violence, violence against women, its impact on the family. It could go much broader than a given yesterday's events with shootings in the crazy amount of gun violence that we continue to see, I think that our office has played a role and continues to play a role and we will step aside from that. We want to address this other type of violence that continues as well. Violence is quite broad. We are still figuring out our space as far as where we want to talk about this.

Thank you very much for coming and for your presentation. You talked about how you looked at the environment as an obstacle to exercise. You also talked about the need to make healthy choices. We all know that there are food deserts and that certain communities don't have access to healthy, fresh fruits and vegetables and other choices that other neighborhoods might. You want to talk a little bit about any plans to address those issues?

F, I think that is a good question. -- Yes, I think that is a good question. This is where I think the champion message comes about we recognize that there is this area where there is huge insecurity which is a huge issue in America. It may or may not be -- food deserts may or may not be contributing to the obesity epidemic. One thing that gives me particular promise, there is this myth that poor people don't want to eat healthy were don't want to live healthy lives. Even if we make it available in schools, kids will throw it away. There was some findings to back that up. But I will give you an example. In New York City there is a foundation that took these innovative approaches, food carts, they did food carts with a go into neighborhoods -- that would go into neighborhoods that were food deserts. They provided fruits and vegetables in a bilingual way, most residents were Spanish-speaking. They found people really desire them the same way that we desire farmers markets. What was happening was in the corner stores, this is also happening in East and South Los Angeles, the corner stores saw that people were eating for the vegetables and so it changed what they were providing to provide more fruits and vegetables because they saw that demand was coming up and that was a profitable market. I think that is one of the roles that our office can play, eliminating that myth that people do not want to have these choices and that there are innovative solutions to get this. You can go to 7-Eleven store corner stores or 7-Eleven's -- or other stores.

Has anyone seen the film, "A place at the table question mark --?" It examines a link between obesity and poverty, particularly with single mother families living on food stamps and how it is impossible to provide healthy meals for children. It goes into all of that. It was news to me but maybe not to all of you. The link between poverty and obesity and other diseases as well. "A place at the table" Is the name of the film. It is a documentary.

I -- have you ever had a chance to visit the Juanita craft diabetes Center. It has many of the features that you described. [ Indiscernible - low volume ] They renovated a school in the poorest most vibrant part of Dallas. You talked about engaging celebrities. We had people come in to talk with gang members and have them have to mitigation together. We have a community garden that is subsidized. With a safe place to exercise. The community is not a safe place for most people. We have quite a large kitchen [ Indiscernible - low volume ] There is a primary care physician there most of the care providers are community workers and nurse practitioners. This is called the diabetes health and wellness assistant, is that the money to Kraft center named after the first member of the Dallas citizens Council -- African-American member of the Dallas citizens Council. [ Indiscernible - low volume ]

That is a wonderful example. I think we would love to visit.

I echo thanks for the opportunity to hear about your work and plans and direction. I am relatively new in this is my second meeting. Last session we heard about the work at AHRQ related to tracking what is happening to people in the use of opioids and the dramatic rise in rates between the ages of 45 and 64. The other thing I am just beginning to see is all of the extra Gary efforts that link with your five areas and especially be notion of championing. Being able to take be type of work that AHRQ has assembled about evidence-based practices. We don't use the word best practices anymore, but best practices. To bring them -- better practices. To bring them as part of your campaigns to the public. I don't know if this is what has already been -- I guess I am directing it to both you and Rick -- what already exists in the way of partnerships, capitalizing on the synergies and how to really make this what it is that I know AHRQ wants to achieve, which is to bring to all consumers, the opportunity to have access to this evidence as they think about making a difference in the five extraordinarily important areas that you articulated. I don't know if either of the two if you could talk about what exists in where you might -- where you think it might go.

Not must yet, we have had a limitary conversation. We are very excited about the directions that the Surgeon General's office is laying out. The clearest area of overlap or synergy is around the opioid work. We are doing substantial work there. In a variety of other areas, as you pointed out Mary, we have been working on trying to figure out how to disseminate information, more to practitioners than two -- to the general public, although we have done some work with the general public. The next action of the agenda is going to be about some transparency with that. We are very excited about trying to figure out how to work together. Historically, at least in the two years I have been here, there have not been cross collaborations. [ Indiscernible - low volume ]

Dr. Bharmal I am with a Medicare QIO. The QIOs across the country are organizing communities of providers and population of beneficiaries together to reduce readmissions and improve care coordination, primarily. But we are also working on medication safety. I think that in organizing the communities, what we have encountered are exactly what you just talked about, prescribing the healthy living lifestyle changes. We are also working on a Million Hearts. The providers have told us the same thing that you have encountered. There is no active living spaces and that is not within their control. In terms of medication safety opioids is one of the -- opioid is one of the classes were targeting aside from anticoagulants and diabetic agents. I am saying there are plenty of opportunities for us to work together. We can help disseminate your message to the community that we are working with.

That is wonderful. Thank you Mary.

[ Indiscernible - speaker too far from microphone ] I had the pleasure of speaking on a panel [ Indiscernible - speaker too far from microphone ] And what employers are doing. We are very unpopular with the cigarette making groups. I can live with that. The employers are focusing on broader well-being. [ Indiscernible - speaker too far from microphone ] A lot of what you said resonates with me because this is what we are doing. I am wondering how we can capitalize on your bully pulpit [ Indiscernible - speaker too far from microphone ] To make their employees healthy and active and productive.

I think that is wonderful to hear. One of the groups that I inadvertently left off the list was employers. That is one of the most important ones. We spend so much of our time at work or school. There was a huge role especially with leadership, that employment come play. That would be great. We should continue our -- can play. That would be great. We should continue our conversation. We feel strongly about engaging with communities. We have actually created a new position that wasn't there before, director of external engagement. It would be wonderful if we could all connect together.

Charlie Roth, I work with the health -- in the national Center for health statistics. I haven't heard anything from my staff about working directly with the Surgeon General. Can we help you in any way in sharpening your message and directing it more -- not appropriately -- in ways that you think might be better served for your purposes. Please let me know.

Thank you. I appreciate that Charlie. All of our campaigns are grounded in evidence-based -- most of our statistics come from the national Center of health statistics in our surveys. What I didn't talk about was, there will also be an evaluation component. We want to see Symmetrix -- some might be helping people. Some might be a different level. Were talking about culture change. It would be wonderful to engage with the national Center for health statistics about metrics and what we have in terms of surveillance to identify if we're making any shifts. That is especially important to me. [ Laughter ]

[ Indiscernible - speaker too far from microphone ]

Thank you. This has been a pleasure. Thank you everyone.

[ Applause ].

It is exciting to see what you are doing. I think the collaboration model with AHRQ and NCHS and the business community and clinicians and other kinds of communities, the thing that we are learning more and more is it is critical for getting success. We have to all be trying to accomplish things together. No one alone can get it done. That exciting to hear. Thank you for sharing time with us.

I am going to leave some cards here so that you can connect with him. Thank you everyone. Enjoy the rest of your meeting.

This is lunch part 2. [ Laughter ] Is that okay?

You are the boss.

I am the messenger. I miss celebrity that delivers the message is. [ Laughter ] -- I am the celebrity that delivers the messages. [ Laughter ]

[ The meeting is on a lunch break until approximately 1:00 EDT. Captioner on standby. ]

In the session I would like to get your advice about what the agency should be doing in the price and quality transparency space. There's a tremendous amount of interest in providing information to consumers and also to providers and to payers about the quality and the price of healthcare. At the last meeting, we talked about the [ Indiscernible ] for value project that we have been working on. This discussion overlaps with that discussion. We are going to have presentations from Christine Crofton who has been at AHRQ for 24 years and is the parent of the CAHPS project, responsible for developing the whole family of CAHPS surveys that we heard from Shari Ling and others. And we will also hear from Virginia Mackay-Smith about a project called MONAHRQ , my own net work and powered by AHRQ . These are two of the main efforts of the agency around public reporting. We have had a variety of other efforts. We have funded -- with CMS money, science of public reporting efforts that started about four years ago that has done some grant funded programs that produce a variety of evidence about toward two more -- how to more effectively [ Indiscernible ] more information. The slide that is a. Which I cannot read and you probably can't either. -- It is selected milestones in the development -- in 1995 with the data released on hospital quality.

[ Indiscernible - speaker too far from microphone ] Excuse me, that was 1987.

[ Laughter ] In the early 90s the AHRQ State Department of health published a report on cardiac [ Indiscernible ] for [ Indiscernible ] surgeons. The founding of the national quality foundation NQS -- NQF was established in 1999. A lot of this was premised on providing information to consumers about quality and they will use it. A lot of that premise has been seriously challenged over the last 15 years. Still, the information -- there is attention being paid from providers and sometimes payers. I am looking for your sage counsel and advice about what our activities should be. I have asked Chris to present on caps -- 11 back and I have asked Virginia to comment on the MONAHRQ efforts . And I am very pleased -- CAHPS and I have asked Virginia to comment on the MONAHRQ efforts. I am very pleased to also have Kathy who has done great work at the foundation. I will stop at those introductions.

Thank you Rick. Let's get this started. What I would like to do today is to condense the 20 year history of CAHPS into as few minutes as I can and give you a brief overview and talk a little bit about the CAHPS survey. I want to talk about the uses of CAHPS data and how that has been use overtime. We will also have challenges and questions for you. I will probably go through some of my [ Indiscernible ] or maybe skip some of them in the interest of getting to discussion later. CAHPS began in 1995. I feel less like the mother and more like the grandmother of this point. There were a couple of things happening then that Proctor the agency to want to fund consumer assessment instruments. A certain part of it was the Clinton health plan which was going to be a regional plan with similar offerings from setting to center. -- Setting to setting. They were also the last of the large-scale purchasers who wanted to have consumer assessment information to use as part of their purchasing decision. There once and instruments out there, most of them homegrown. There was not a lot of confidence in the reliability and validity. Into QA -- NCQA had an interest in developing a survey that would be developed across the board and would the reliable and valid. That was our starting point. At the beginning of CAHPS we only ever planned to do a health plan survey . That is where we started we developed design principles as we went along. Some of them are here. We have a fairly extensive set. These are ones that I think are important. The first one is that we vote is the topics and questions on every guess for which consumers are the best or only judge. Understanding the instructions your provider gives to you is important. That will affect the outcome of your care. That is a question that only consumers -- only the patient can answer. If your provider did or did not spend enough time with you, again, if you don't get all of your questions answered, that will affect your outcome. That is another question that only consumers can respond to. Along with that, we wanted to produce questions that were ratings and reports rather than satisfaction questions. That was for a number of reasons. If we have reports of patients's experiences those items such as digital Doctor treat you with courtesy and respect are more actionable, understandable, object and specific then general satisfaction ratings. Also there is an element of expectation in responding to these ratings. If you are a person with high expectations for your healthcare and those high expectations are met, you are going to rate satisfaction quite highly. If you have low expectations and those low expectations are met you might also rated highly. We wanted to give solid information about how to make healthcare decisions. One of the questionnaires -- we wanted it to produce data is valid and reliable as possible. We did a great deal of testing to get to that point. We started with cognitive testing. That is a method for making short that -- making sure that the respondent on the survey understand the question and the response options in the way that the developer intended. That might consist in sitting down with a target audience and asking the think aloud of the answer questions put you may have a person complete all survey and then debrief with an interviewer. You don't always find out that you have hit the market first time. This is an iterative process. You may have to go back and get out and test a new item. We also field test all of the service. This gives us an opportunity to see how the administration instructions work and to see if we need to straighten them out before we publish them. So people can use them. It also helps us determine the reliability and validity -- you can do factor analysis to see how items cluster and whether they belong and whether composites need to be reworked. We also have a a great deal of stakeholder input throughout the process. It starts with periods of public comment that we did even before we work required by OMB to do it. At the beginning, we published a general idea of the survey, what we are doing and why we are doing it. Farther along in the process we published a first draft and got comments on that. I won't go through each and every example. We tried to reach as broadly as we could in getting feedback about the survey because we get expert help in areas in which we are not experts on top it's -- on topics such as mobility impairment and dialysis facilities. Is also a good way of developing championships, people who are willing to get out there and provoke the survey -- promote the survey. If you look at the literature on change, in any setting, business or education, you find that if you include stakeholders from the beginning, the process of getting not survey or innovation to take hold is a lot easier. Rick said something about my responsibility for CAHPS . The fact is CAHPS is a huge team and we have a great team. Right now it is the AHRQ CAHPS force. There is a cap user network. They run the technical assistance line -- 11 Matt user network. They run the technical assistance line and we owe a lot of thanks to them. We also have a group that collaborates financially and intellectually on the surveys and they are included as far as the team as well. The surveys themselves, removed from health plan survey to quite a variety of facility and ambulatory surveys, there are a whole bunch of them. Rather than go through them, I will mention this. On the last page of the slide there is a URL for the websites are you can take a look at our whole inventory. When you look at the surveys come of the fact that a lot of them are prepared for different settings -- surveys, the fact that a lot of them are prepared for different settings, we have well over 60 surveys that we have developed and we had maintained over time.

CMS is one of the biggest CAHPS users . We have a lot of other users as well, OPM which collects data to assist retirees and government [ Indiscernible ] with government health plans, purchasing coalition such as Massachusetts health quality partnership. Lots of organizations for lots of different purposes. We will talk a little bit more about that in a minute. It is hard to track the number of people who do use CAHPS surveys because confidentiality requirements means we cannot track them as they come to the website. They can use the survey for free. With the data that we do have we know that over 146 They can use the survey for free. With the data that we do have we know that over 146 million Americans received care through plans or facilities that collect CAHPS data . CMS has a lot of involvement in CAHPS . I wanted to talk about AHRQ and our unique role in CAHPS . I would like to get us to the discussion as soon as I can. I am not going to go every other item. If you have a standardized survey, you have to have an organization that defends the standard. You have to have a point that? All of the information about where the problems are unaware changes need to be made to, that updates surveys that does additional testing. That organization is AHRQ . I think that is our biggest point of uniqueness. We also hold the copyright to the CAHPS trademark . We can't go around and police organization to make sure that you are using CAHPS the right way. But if we find them, we can tell them that they are free to use the survey any way that they want. If they want the CAHPS trademark on it, they need to use the survey as written and follow the administration sampling rules and all of that stuff.

This slide might better have been titled evolution of CAHPS . This isn't all of the great marker points in our history but it is a good sampling of them. We began with health plans. We moved pretty quickly after that to developing surveys in ambulatory care settings, and other settings. Other agencies began to use it for research. -- [ Indiscernible ] policy researchers began to use CAHPS for quality -- improvements quality of care. When the Internet came that caused a huge shift in our ability to get CAHPS data out to people . CMS began publishing hospital compare, home health compare, and a lot of others. As Rick said a lot of organization started to publish consumer assessment surveys. Organizations used CAHPS in advertising. I was watching TV last weekend and me answer treatment centers of America was bragging about their CAHPS . It's all over websites. CMS another private organizations are using CAHPS data as a basis for reimbursement. That is a very high-stakes use. It has caused renewed and increased scrutiny of the way that we develop the surveys. With that, I have included an example of reporting CAHPS data from the quality and disability disparities report. It has a new name. I will leave you to look at that your leisure. When we were putting these points together, we went and asked if you use CAHPS, what are the bumps in the road . What are the problems? If you work in an organization that collects CAHPS data that you are not directly involved, what are you hearing about CAHPS ? We also wanted to know where the challenges were and how we could better address them. To get started, some of the challenges, I think Rick already mentioned one, has to do with consumer use of CAHPS data. At the beginning , we thought this is going to be a winner. We have got validate to, a good way to report it to people. People are going to climb all over this. There are certain situations where people are more likely to use CAHPS data. If they have moved into a new area and need to find a new health plan or employer or if they have a new employer and need to switch to a different healthcare plan -- the question is is CAHPS really important. Should we keep pushing for that personal individual use or are we getting to the same goals via quality improvement and better services for consumers via other paths -- or via other paths. Despite our best intentions over the past 20 years, there is still a conversation about patient experience versus satisfaction. Since organizations are making high-stakes decisions based on CAHPS data , people want to know that this information is solid. They sometimes feel confused. Surveys were satisfaction surveys -- are satisfaction surveys -- CAHPS surveys are satisfaction surveys and they don't consider those enough. They expressed concerns that CAHPS is a survey developed by researchers for researchers. At a recent meeting at an organization that while -- I will not name, one person asked to develop CAHPS . The respondent talked about that. And the person said, ivy tower researchers. I think we had a huge range of input but apparently we are not doing the best job of getting that information out. That is another area where we could use your feedback. Another continuing one is practical concerns versus mythological -- method two logical rigor. There is a lot of pressure for us to move to an electronic administration mode. Were testing that. I assume that is the shape of things to come. There are problems with developing a proper sample with that method and other methodological problems. Giving a person in a physician's office a tablet may not be the best situation for filling out a survey. A lot has to be developed before we move in that direction. We can't sacrifice the methodological rigor of the surveys given that there are these [ Indiscernible ] uses. I will stop talking and see if you have questions or comments or anything else to say about CAHPS .

We were heavy users of the CAHPS family survey at Kaiser Permanente. Would always struck me as funny and generally [ Indiscernible ] some of the hospitals and clinics within our own systems and only used CAHPS because we had to and had the organizational commitment to it but also spend money on other survey organization's tools -- organizations' tools. Why are people using them and are these functions good or should they be filled by CAHPS or not?

That's an interesting question. First of all people can add their own questions to CAHPS surveys. We ask that they -- when people complain about the length of the survey, sometimes it is because they add a stream of questions after the CAHPS COR questions. That is an issue that I am not sure of how to address. I think one way is to develop additional supplemental items that we can test to see how few items we can get to address certain concepts. That might allow users to use the questions. If we can show them that they are getting the same breadth of feedback with you are questions, that might be helpful. Another issue -- I am not sure if you are focusing on this -- there are many CAHPS surveys . There is overlap across surveys because we do use a set of COR items . There are surveys that are based on the clinician group survey like accountable care organizations, patient centered medical homes. There are multiple requirements for multiple organizations to use one or another version of that survey. If you have got to field a survey with a slightly different sample, that is a slightly different survey. This is considering -- light -- this is a slight concern. I don't know if they have to report to different organizations for accreditation.

I was wondering what concept of domains you are trying to -- or domains you are trying to get out in the surveys?

As an example why would hospitals use H-CAHPS and other surveys? That was the intent of the question. In that case --

In that case, we got great pushback when we began developing H-CAHPS. I think there was some concern that having this instrument available for free -- it would soon be the standard and required by organizations by CMS -- was not an activity that the government should be doing. It may be that [ Indiscernible ] wants to add questions to the survey or do a different survey in an effort to boost market share. To be fair a lot of organizations were using [ Indiscernible ] before H-CAHPS came on the scene. They may be questions -- there may be questions -- maybe organizations have more of an opportunity to frame their questions. Those are my suppositions but I don't know how well they meet the circumstances.

With competition analysis, do you look at what the other surveys are doing and try to decide whether that is within your purview are not ask I know one of the differences is that it is more supportive to do those other kinds of surveys on much more -- at much more frequent intervals to try to drive improvement and not just once a year. That's a little bit warranted with some of the CAHPS things.

I think that is an important concern part that came into -- came into effect when we started doing quality improvement. People want rapid cycle service. That is something we talked about on the team. We are trying to address this in what ways we can but I can't say that we are totally successful with that. That is something I can take back to the team and discuss with them about in more detail.

[ Indiscernible - speaker too far from microphone ] People run circles around me with their knowledge. Whenever I have had a question I always called Bob Kaplan. [ Laughter ] My question is more of a strategic question getting to what Rick was saying before. We have developed something that is so successful and meet such a need that it has now created other problems, I think. It is being used in different ways that you never intended. I am thinking, what are the implications that has for CAHPS itself ? I am not well-versed with this enough. There needs to be an ongoing -- two things -- one, an ongoing assessment. It is an excellent instrument but doesn't perform equally well everywhere. As you get more experience, hundred and studies identifying this problem or that problem or whatever? Do you have a formal way of tracking that? With that I think there should be some investment -- not a lot of money -- by the agency in supporting people to continually test to try to improve the skill. This is one of those things that you can't change every time there is a slightly better way of doing it. It's not like a piece of software where you upgraded. There needs to be consistency. To do something consecutive -- significantly better, a change may be necessary. That is what they pay you the big bucks for.

[ Laughter ]

I do think that there would be value in formally evaluating how it is doing for what it is being used for two reasons. One is to figure out where it needs to be improved. Two, to issue warnings to well-intentioned people on the outside who were using it for something it is liquid you do a good job on. The other thing is, for research -- [ Indiscernible - low volume ] -- We understand if there is a correlation .64 an item or scale or something like that -- for an item or scale or something like that. That is a big picture. It may not be fine for an institution that is getting paid that we. That may not be a bigger -- big enough error rate that it would cause problems. How do these things track over time. [ Indiscernible - low volume ] Who are studying this and seeing how it is working to figure out how to improve. I guess there are three tiers, telling people who are using out -- using at how not to use it. [ Captioners Transitioning ]

I guess the earth three tiers, evaluating and feeding back to people who use it how not to use it, not just where it should be used but where it should not be used.

I think you have a cluster of really good suggestions. Rick has talk to us about the possibility of developing an evaluation and I think it is high time. I think there are lots of questions that could be answered by looking at the past trajectory of CAHPS and seeing where it has been successful and not so successful. And I do not think that the CAHPS team should be the ones to do that, they know more about it than anybody but you need to have someone who is somewhat removed to be able to look at those things. You're right.

No one has ever said that before. That is any statement.

[ laughter ]

It is a good day.

It is an issue for us. In terms of budget we do not have big-box and I think that the CAHPS budget has been static.

What I was thinking here, and Rick you can tell me if this is legal or Jamie or Sharon, this might be an area where they can bring together some consortium funding, I am thinking maybe payers and hospitals, insurers and hospitals might be willing to pull some money with the agency to address issues about how you use this for reimbursement. Because payers and insurers and hospitals or health systems will be oriented. Maybe it is one of these comments, it needs to be done and everybody knows it, nobody wants to kick it in but everybody also benefit from as well.

There have been some NIH studies and multicenter studies were manufacturers have kept - have kicked in some of the support because they want the same questions answered. I don't know if it is legal or if you have the authority to do it, but considering the budget and other resources, because it is now being used for reimbursement maybe even CMS would be interested in being part of a collaborative private/public sector consortium that would allow the other do this with you guys coordinating it in organizing it and taking the lead on it.

We could certainly look into it. Our lawyers who are not in the room will tell us whether it is legal or not.

[ laughter ]

We should move on. As we are moving on to the discussion of MONAHRQ and Kathy's comments later, one thing I have been struggling with some, and I look over to comments but not right now since we do need to move on, is this question about consumer use. There is not a lot of research, but there is some since at least the to the extent that consumers are paying attention to information about providers, it is more likely to be health grades than it is to be NQF endorsed measures, including CAHPS.

Kristin talk about whether there is a component and Mark has been using or or he on the use of narratives or adding the use of narratives to CAHPS and coming up with some very positive and optimistic findings. But as we think about this going forward, a big part of the question I have is that and where should we, as an agency be if we are trying to produce information useful to consumers, a big investment in CAHPS which is used as you have heard and useful in part, but maybe not what consumers are really wanting. But having said that I will not let you answer it yet. I will turn over to Virginia Mackay-Smith to tell us about AHRQ.

Okay, everything is showing up where it is supposed to show up, thanks Rick, I am Ginger and I am temporarily the director for the Center of delivery organization and market while we search for replacement for Irene Fraser. Even before she left though, I was lucky enough to service the senior advisor on the MONAHRQ project. And Rick just talked about sort of the lay of the land of the development of public reporting and MONAHRQ is AHRQs effort to join in that conversation, once you have the data in the measures, how do you go about moving the data out of the offices where the data are collected and into the public? Where is the button? There it is.

So, the star of this program was about six or seven years ago at a point when there were a lot of new measures and data being developed and collected around the world. In addition, here at home at AHRQ we were encountering some organizations, our state partners from AHRQs healthcare cost and utilization Project and these are organizations collecting their own hospital discharge data and were interested in ways to be able to report those data easily and appropriately to the world.

At about the same time there were a lot of states that were beginning to develop laws and mandates for public reporting. There was sort of a growing chorus out there. At about that time a lot of those organizations and states started discovering a problem in that developing an interactive website for public reporting turns out to be very hard to do. It is expensive and it takes very specialized expertise and it seemed to be a big waste to have different states and organizations developing their own software and reports. And so AHRQ thought how can we contribute? Our idea was to develop a public reporting software that anybody could use. The goal was that MONAHRQ would be a reliable tool to lower the threshold creating, launching, and hosting a public reporting site.

And the way that it would be reliable is that we are AHRQ, we basin on the evidence and we draw on the latest developments in the science of public reporting, and we would facilitate access to the best data and to measures that have been tested and used widely. So that is our goal. Early on we decided that we needed some guidelines if we are going to be a reliable tool that others could use.

So we have these principles for us. And MONAHRQ has to be evidence-based and by that we mean we have to use established and vetted measures, reliable data sources, and techniques that are shown to be effective in public reporting. It is quick, this is my favorite one. Some of the earliest states that develop public reporting sites before MONAHRQ spent months developing that software. Today you can Del Valle - today you can develop a MONAHRQ site in less than one day, and we can prove it at a public contractor assigning it to an assistant is that here's a software, download that, grab the data from CMS and build a site. And he did it in less than a day.

It is also easy. It is designed to be user-friendly and readily follow bowl. There are user guides and decision guides to bring you right along the way as you do it. We are also worried about the one size does not really ever fit all process. So it is also designed to be very flexible and customizable, if that is a word, I get nervous sitting next to the sky and using things that are not words.

[ laughter ]

That's okay. [ Indiscernible-low volume ] [ laughter ]

Most importantly, well, not most importantly, but importantly MONAHRQ of course is also free. That was a significant issue for states at the time. And thinking about Sandy's question I think it continues to be an issue when we have a free software and a marketplace where there are also proprietary software products available. So when you are asking Chris that question I thought yeah, that really applies to MONAHRQ as well. The software is free and you can download it from the website.

In addition to the software being free the technical assistance is free as well. And that turns out to be a real boon to our users. So here's a brief history on MONAHRQ, and like Chris I think it is more of an evolution really than a history. So MONAHRQ started as a project under AHRQs HCUP program. These programs were HCUP partners and there were a small handful of them in the early days he worked closely with us and developing the first versions of MONAHRQ. That core of initial users became critical partners in providing feedback on how the software was working and suggestions for what else needed to be added.

So the first version of MONAHRQ was launched in June 2010, MONAHRQ 1.0. That version included 57 of the AHRQ quality indicator measures and it had the capability of producing County-based rebuttable hospitalization maps. Those are two of the early selling points. 2.0 was launched a year later and it included almost all of the AHRQ quality indicators and also for the first time included some CMS results, CMS compare results.

Then we were rolling. Over the next couple of years we developed more versions and we just listed 3.0 and 4.0 but there were some interim versions as well. And again with each one we were adding new measures and updating the data and adding new web elements to the software. We also were adding a lot of improvements that have been suggested by the host users as we went along.

In 2012, AHRQ and MONAHRQ reached a turning point, more states were creating mandates and we had the first few versions of MONAHRQ out in the field being tested and used. And so MONAHRQ graduated from being a project under HCUP to being its own standalone program as we were awarded a five-year contract dedicated to enhancing, refining, and disseminating the MONAHRQ software to the world, to the nation.

The second big thing that happened in 2012 is that CMS formalized their collaboration with us and they had been very interested in working on the early ideas of MONAHRQ with us. And we entered into a partnership about that time your which included the public reporting grants that Rick had on his timeline. And it also included a commitment to share the cost for four out of the five years of the MONAHRQ contract and so CMS has been instrumental in the development of the software.

So with this contract, as you can see, contract is been in place three years now. And as MONAHRQ has grown so have the number of state host users as well and as you can see from the chart it has tripled in number from the beginning of the contract. So I wanted to tell you a little bit about those states. The thing is, you can download MONAHRQ directly from the website and so we do not know when you do that. We have no idea who is downloading the software and who is building a site, unless they call is for technical assistance and then we have an idea who is there.

We also, and by we I mean our contractor, goes out looking for MONAHRQ sites. And so the 15 dark blue sites on this slide are the 15 states that we say have a public reporting MONAHRQ site. We tend to say there are at least 15 sites because we hold out the possibility there are others out there we just have not found yet.

Now, the green sites, the green states also have a statewide MONAHRQ site but it is not a public report, it is a private report and these sites are accessible only to the host organizations, staff, or membership and so they do not count as a public reporting site and they are not available to consumers. It is our understanding from the kinds they are asking in TA they are looking at this from is the quality improvement purposes and data verification.

And I also wanted to point out the two aqua blue states on the West Coast, these are states that have downloaded the software and have called us for technical assistance, and we have been working with them as they experiment with the software and their data. But they have not yet launched a site. But we live in hope for that.

These are the state host users we have. Again, based on the kinds of technical assistance calls that we get there are other kinds of host users as well. For instance, there are quite a few academic and research organizations that have built MONAHRQ sites for their own use. Primarily we think for research purposes. And interestingly, there is a handful of consulting firms that have downloaded the software and built sites and again, we know from TA calls they are using the software to run analyses and reports for their clients. The it is a nicely versatile software package.

Know I said that building a MONAHRQ site is easy so I will prove that to you by showing the steps on one slide. If you want more details I could direct you to a seven minute video on YouTube or if you are a real glutton for punishment there is the 200+ page users guide that you can look at as well. Step number one, download the software. The software and instructions and other tools that you will need such as the AHRQ quality indicators software are right there on our website and available.

Step number two, upload your data. Ordinarily estate will load their own hospital discharge data, emergency room data, and so on. Step three, load the measures. We use CAHPS and we use hospital compare results and of course the AHRQ UIs are a major part of the site as well. Step number four, make the website your own. There are lots and lots of opportunities to customize a MONAHRQ site to make it to the purposes and intended audience of the site host.

Step five, bingo. You have a site. The MONAHRQ software has quality control checks built into it, consistency checks, and the host user can look at a site on their own before it goes live to make sure it is exactly what they want. Then they are often running. Now that you know how to build a site without you might be interested in seeing what one looks like. Now I have to follow my instructions here.

[ Indiscernible-muffled speaker ]

I did it wrong didn't die? - I did it wrong, didn't I? I forgot to click on the URL first. This is why we had Alfred.

[ Indiscernible-low volume ]

Is this going to work? Are they seeing it as well? Oh, good. So, this is the Utah site. Utah, by the way, was one of those core group of initial MONAHRQ sites and it is also one of the states that have launched the MONAHRQ 6.0 site which was just launched a couple of months ago. This is their homepage and you can see right away one of the features of customization, there they have their own name and logo right up top. They can also use our color scheme and are look and feel, this graphic I am happy to say is ours.

Just to give you a quick to her of the homepage, - a quick tour of the homepage, there are the navigation tabs at the top of their the tabs in the body of the page which turns out to be best practices for Web development and is easy to use for the uninitiated. We want to also do some education of the consumer user as they go along and so we include some of what we call guided education which is this kind of content here on what is quality? You can use ours which is vetted by our technical advisory panel or you can substitute your own text in there as well.

We have also developed new with 6.0 a video that is a step-by-step on how to use the site. And we are also including new educational tools like an info graphic. We are pilot testing it with this one in 6.0, we only have one, you see how it goes in and learn from the experience and add more in the future. And down here at the footer, you can see there is a little plug for MONAHRQ on the left and also an area where the host user can add information about who they are. And that is sort of the homepage. The real point of this is what can you do with it?

Let's start with browsing for hospitals. When you click on rows you come to this page. Right now the default is you go to hospital. You can also search by condition or topic. But for our purposes today let's search by hospital. Please search by hospital. Okay. We will go by hospital type, let's say general and we will run reports. Now this is the Utah site so right now I am getting a report that includes all the general hospitals in Utah. There they all are.

Now here is a little bit about best practices for consumer report. It turns out that symbols are very helpful, especially when a company has a little explanation of what the symbol means. So you have got the list of the hospitals, the general hospitals, and their score. The score is under what the measure is, how patients break the hospital overall. If you want to know what that means just click on it and MONAHRQ will give you an explanation of what that measure is, along with inks down here for the more sophisticated user to get more information on that user or measure.

Let's go ahead and compare some of these. I picked a couple that have good information.

[ Indiscernible-low volume ] All right . I will move it along. It is easy to get distracted with this. All right. We will compare some hospitals. You can compare them to the national average or to the Utah average. You can use symbols. Let's take a look at some childbirth quality measures. Again here the measures on the left and here are all of the quality scores by hospital. If you see a hospital that you become interested in you can click on it and get some profile information about that hospital. It will also give you all the scores for the hospital as well.

You can do similar things for nursing homes and for physicians we have profile information, we do not yet have quality information in there for physicians. We are still exploring the best way to display those. Now to get back to where we started?

[ Indiscernible-low volume ]

Alfred? [ Indiscernible-low volume ] good. We are back. So this is just a slide that summarizes some of the new speech or's that we have put on MONAHRQ 6.0 and it is a major step forward, all of them so far have put it hospital reports and as you just saw the Utah site this one adds nursing home, position profile, and also interestingly the ability to show trends over time which is something that most users have been asking for a long time. Next will be of course MONAHRQ seven. These are the types of features we are considering for including and it will be a long conversation. And given how enthusiastically that MONAHRQ has been adopted by the host user so far we will also step up the dissemination efforts for MONAHRQ seven and try to extend it to more states that have mandates.

As Chris said we are interested in your feedback on this as well because we are always looking to improve MONAHRQ and make it more appropriate to the users, as well as to the field.

Yeah, I very much look forward to the feedback. We are conscious of time as well. I will introduce Kathy Hempstead. [ Indiscernible ]

So as we talked about this meeting without it would be great to get perspective from somebody outside of AHRQ and Kathy I am pleased that you are able to join us today. Is the team director and team ups are at RWJ leading their work on coverage and now that we have seen that she has gone on to work on transparency and value and a bunch of other things. In the interest of time I will not go through all the reasons that she is highly qualified to talk about this, I'm just when to let her prove it by talking to you. Take it away Kathy.

Thank you very much. [ Indiscernible-speaker too distant from microphone ]

And you are building a site for which states?

[ laughter ]

[ Indiscernible-low volume ] our staff had the ability to travel to Utah to learn all of this elaborate SAS programming [ Indiscernible-speaker too distant from microphone ] does that will not work? Okay thanks. That is just a little bit of a reaction to the MONAHRQ presentation I come to you from a foundation who is a huge fan of AHRQ, I wanted to start off by saying that and at the Robert Wood Johnson foundation we love AHRQ and we feel like so much that we are trying to do the same things which is add useful information to improve policymaking and decision-making your we have a tremendous amount of respect for what AHRQ does and we see her expertise and it was on display with these two presentations. It is really, really excellent creations of metrics and measurements, very high quality data collection and an enormous amount of effort to both analyze and also disseminate the data widely, which you can see both in CAHPS and then also in MONAHRQ which is a vehicle for people to disseminate lots of data.

I thought those are two really good examples of the kinds of things that AHRQ can do and we are involved with a project with AHRQ right now that was not talked about here which I think is incredibly exciting and that is to add divider organization information to the maps is a module of the maps. I ultimately think that will be such an important data source for us to use. In fact I cannot imagine a more valuable data source for people who need to be able to evaluate how these changes that we are making in payment and plan design, what kind of impact they have on cost and utilization? I think that assists another great thing that AHRQ is really well set up to do. I know there is not much time left and I deftly want to hear whatever but else has to say, but the idea is thinking about what is AHRQ doing in the transparency space and what more can they do? To me it seems of these two projects are really important and Austrian transparency, but maybe not so much by being heavily used by consumers to buy being used by people further upstream in the delivery system.

I actually think that is very important. I think the kind of things that CAHPS measures are very important and they are definitely are the way care should be delivered, and things that patients should get when they receive care but I can understand why consumers might not be super interested in those measures themselves. But I think consumers definitely benefit because the measures are collected and because payers are looking at them and regulators are looking at them and it makes providers be accountable, even if they are also collecting other firms or forms of survey data and so I think it plays an enormous role.

I think the idea of evaluating CAHPS, not the anybody asked me, but I think it does seem like a good idea. If there could be a way to speed up the administration time and reduce unit cost by changing to some other mode of administration that might increase the usefulness of the survey. You know, MONAHRQ, those sites are beautiful and I cannot understand what any state would not download the software and build a sites I think you should do some research on that and find out what are they doing instead. A lot of those sites, a lot of people and up saying consumers don't really use them and so on. But I do think that a lot of very important stakeholders do use those sites and look at them and I do think that consumers benefit a lot. In those respects, those are definitely projects that foster transparency and contribute to transparency.

I thought I would just say quickly a couple of other things to maybe make people think about some other things that AHRQ may or may not consider doing in the transparency space. I was struck by what Rick said, and I totally agree, he said something like to the extent to which consumers pay attention to quality information about providers it is more's like - most likely will be Yelp kind of stuff. And that is a good point eared we give consumers a lot of information about providers and facilities that frankly is a little bit land and boring and it doesn't grab them and it does not compel them to make any kind of decisions.

I still think it should be collected and it is important but it does not really motivate them. I think one way to look at that is to say maybe that is okay and maybe we have regulated health care enough so they are not really these important decisions consumers need to make your some times I think about the analogy to the airline industry where we do not really compare safety of major curling carriers anymore. We just think about our customer experience, which we all hate, but the relative aspects of that.

[ laughter ]

I am not at all sure that is true. I have another grantee that is very different Than AHRQ and it is Pro-Publica an if consumers maybe didn't know things they did know now they would do things differently but I don't have anyone has been following the brouhaha over the surgeon's scorecard which was just released which on the one hand I think interests people very, very much. This is the kind of thing a consumer would want to know, if I am going to get surgery who should I not go to? This is a very salient issue. But on the other hand a lot of people criticized the methodology saying they have not really made valid comparisons and they have not really risk adjusted. And so I think that Pro-Publica steps into this void between Yelp and very sanctified but bland quality measures.

I was thinking to myself why isn't AHRQ doing something like that because if AHRQ had constructed those measures they would be more unassailable and maybe the conclusion would be this is not good and this is not valid, and if there were important differences it would be more defensible. And I think that considering that Pro-Publica are journalists they try hard to do a good job and they are not just making it up in an afternoon. They work on what they do for a very long time and they are very transparent in their methodology but they are saying this is a first step and make it better if you want to or if you think you can.

It occurred to me with AHRQs expertise and the ability to create really valid measures, and a sense Pro-Publica is teaming up with CMS to put out these measures and I think that would be a cool thing for AHRQ to work on and it would have a lot more credibility and gravitas if AHRQ did something like that to the extent in which there are some important areas to look into where quality difference do matter.

The other thing I want to say quickly is I think there is a new arena of quality issues that are kind of at the planning level and more and more people are choosing plan level attributes, and I don't know if CAHPS is really capturing it, but I think we need to make up some new metrics. And I think that that is something that AHRQ is very good at. And one of the more obvious things, though there are others, thinking about how to objectively characterize people's provider networks. Talk a lot about narrow networks and there are these very crude classifications, HMO, PPO, but I do not think they are super accurate. I do not the consumers know what they mean and we are asking consumers to make a trade-off basically between price and access to providers and it is a trade-off I think consumers should make with full knowledge of what they are giving up and what they're getting in return.

So plan choice decision support is a very important issue of mine and I have definitely worked closely with Rick and other people on sponsoring opportunities for developers to build these different kinds of decision support tools. And that is great. But what they are doing right now basically is organizing existing facts into fact patterns so consumers can understand what they're getting or whether Coster going to be. I think there are some higher order things that AHRQ to play really important role in and out would be to create some new measures that would be meaningful to consumers and then disseminate them so consumers understand them so when they are choosing plans they are these understandable plan level attributes that people could be looking for as well. I'll just stop.

Could you give a couple of examples of those things you're thinking about?

I think one of them is the network. The characteristic -

Did you have a medical problem and more you able to see the specialist that you want?

I think an objective classification of your network size. You might think that you are buying a narrow network, how narrow is it? What percentage of specialists within your MSA would you be able to see if you needed to? Even this is harder than it should be, to be able to see some particular provider they want to in the network but if they are buying a narrow network plan I don't think people know what that means how narrow is narrow?

Were you told the provider was in the network but then it turned on the provider is not in the network? Or what your doctors are not part of the network at the hospital in the network.

[ Indiscernible-low volume ]

Or even things like what kind of step therapy do you use for a certain kind of prescription drug? There are a lot of things behind the stage right now, and some of it is too much for consumers to have to deal with, but I think there are some plan level attributes that consumers should be more aware of when they are choosing plans. And I think that AHRQ has a lot of expertise in insurance and measurement designs I think that would be a really useful wall - useful role for you to play. Thank you.

Sure.

Can I ask if you look at any of the commercial tools are available and in thinking about where to go from here I know we have several members in the data transparency space that are identifying areas, let's say MRI facilities that are a fraction of the cost for the same quality? Or identifying centers of excellence for certain types of care? Or, if for example, if you type in diabetes up pops some education. So it will get you to a primary care physician for me like with back pain or immediately move towards the surgery route. Also the costs. I have the pharmacy site is particularly complicated because the rates change everyday. I know that the transparency vendors are having trouble with real time, her store costs. But that is where they are heading as well because that is something, those are the kinds of things that consumers can shop for everyday.

We have primarily been involved with tools that allow people to choose plans and we are interested right now in three different topics that correspond to places where people had a lot of problems. One is the most simple which is sort of the out-of-pocket cost calculator people could enter a little bit of information about themselves and then be able to get an estimate of how much they might have to spend in a year, a good year, bad year, middling year, so people can look beyond the premium to see what is my out-of-pocket?

The next project we are involved with, and we're working with Rick on this, is a provider directory project to address the issue that you brought up that a lot of times people are not looking at accurate lists of providers and do not have a very easy time figuring out whether or not a physician they want to see is in their plan. It begs a bigger problem which is that people have very few resources to choose a new physician which gets me to a whole different problem that I think is really important as well and that is people cling to physician sometimes they do not have good tools to find another one.

But be that as it may, I think the provider directory problem has been significant but I think CMS is made huge strides and requires much more disclosure from plans to think what help a lot. The third piece we're focusing on is the prescription drug aspect of planned choice and making people understand what the drugs cost, or the formula to drug tears are, what the therapies are, and whether or not they can identify whether certain plans are tearing away for certain diseased states. We are focusing our work in the planned choice arena right now. But I think there are challenging transparency issues once people have a plan as well. Carriers build a lot of tools and there are a lot of needs for information there as well.

Catherine, this question is for you. I am really interested in where consumers can get information like that. Do they just go to the Robert with Johnson website? How would a consumer get your information? I think you are absolutely right, particularly people go to the healthcare marketplace is now. There is a lot of information there and it is certainly much more helpful than having to call every insurer and find out what their plans are and get all of this advertising stuff in the mail. But you know, being associated with one of the state exchanges, or the federal exchange, would be a great place to have this kind of tools.

So what we have done is we have sponsored some challenges for apps developers, we work with this company called Health 2.0. We will talk what the first challenge, three companies that were pretty well-known, consumer check which doesn't start for federal employees, Stride Health which is a web-based broker, and Clear Health Analytics which is more of a new entry. All of those companies want to get business. And so they pitch themselves to the state exchanges and I think they have pitched themselves to the feds. Stride exchange products directly but would like to be able to sell exchange products directly but I think the market is emerging to meet this demand that people have.

My own view is that everyone should have access to at least one of these planned choice tools to hopefully someday, in my own vision is that these sort of advertisement supported a sort of freak to consumers to use so people can look at a few different sources of information. I do think the prevalence of having to search directly for health insurance plan is rising really quickly. I think these decision-support tools are really important. I also think that plan design needs to get simpler but in the meantime people need better tools.

Terrific presentations. Thank you all. The first thing I want to say is I'm headed to Utah the next week or so. We clearly have not heard of Lake will be gone, everybody was below average so I will stay in great health. So with MONAHRQ obviously a metric of success is going from a couple of states to 15 states and terms of transparency and public reporting. I am wondering, have you thought about the kind of assessment of how the availability of data across sites in the transparency of that is influencing health transformation in those states?

That is the real issue for us. The thing about MONAHRQ is it is a tool for states to use but our focus is on what do the states want and what do they need? And we tried to salt the mind and say that part of what you might want and need state is to understand how your end-users are benefiting from this data and how they are using them in their own health decisions and so on your that we have to go around our elbow to get there.

It seems like you have some extraordinary, even just case studies, about how multiple users are taking advantage to collectively accelerate improvement.

And you are right. That is what we have so far. We do have a few case studies with our low-end. And we have some focus groups and key informant data from talking to users who we then walk through a site to say and? What next? Interestingly one of the new bits of information that we have gotten is that the consumer advocate groups tend to be very interested in this kind of a site. It may be that this is a target audience that we will be focusing on as well if we can get to them, and they can get to the consumers.

Thank you, that was great.

Other questions or comments for any of the panelists? Mary?

[ Indiscernible-low volume ] [ Indiscernible-speaker too distant from microphone ]

In terms of measures the CAHPS survey is really the predominant measurement for the area. One of the goals in the area is to improve shared decision-making. And I know that you have a few questions regarding medications, that your doctor talk to you about taking these medications and so forth. I do not think that it gets to the issue. There is no question that the final decision made incorporates your preferences or your goals. Will you be working more towards that are developing more questions regarding shared decision-making?

We have been working for so many years on that area. It is very difficult to develop papal and pincer - paper and pencil survey questions that can manage that. It is hard to parse out into different components and that is one area of questions where we have done cognitive testing and we come back empty-handed. We get a range of interpretation of questions but there isn't any solid piece to hang onto that gives us direction to go. It is a very important area and it is not something that we are giving up on. But I think that we maybe need to develop new and different tools for approaching that. That might be the kind of information that can only be obtained an interview setting.

Rick mentioned our working on narrative information to include with the ratings. Maybe there is a way to look at the narrative information a provider or hospital or plan gets and analyzing that and see if there is information in there about coordination of care. But that is something that is incredibly difficult. We are not giving up.

And Mary, on shared decision-making I think you are exactly right, it is a really important area, whether that needs to be procedure or condition specific, you know if you are being recommended for a specific for seizure was sort of information did you get and where your preferences were solicited or could it be done in the more general framework? And if it does need to be procedure or condition specific that is another level of difficulty. It does not mean it is impossible but it is much harder to do.

The question really pertains to medication which I think is broad, and it is good, that it did not have the additional question about the final decision regarding this medication Incorporated your preferences.

I was curious, what is going on with all the payer databases? I think that our WJ had a role in getting this things going and is that data available to MONAHRQ to pull upon?

One of the things we just did this spring was released something that we called the open-source framework which is a user friendly template of the MONAHRQ software code so that most users can build in their own data sources. And we had the APCDs in mind when we did this because that data is state specific and it is not a national data set. We just released that and we are working through TA, with one or two?

[ Indiscernible-low volume ] With four states are exploring doing precisely that.

We have a grant that one of Gingers colleagues, the project officer or program officer with Stanford with the subcontractor of the [ Indiscernible ] data counsel working on the development of uniform measures that APCDs could use to report on both quality and resource use. You know, one of the things that I learned from Ginger and colleagues is part of how HCUP got to have relatively uniform data across states was through the development and implementation of the quality indicators. So once the QIs were developed and states apply them to their hospital discharge data and they started realizing this is really crazy stuff and it does not look the way it should, that led to much greater uniformity in the data. And the thought around this grant of measures for APCDs would move in that direction. Sort of the fundamental question is the extent to which useful information on quality and price will come from APCD data or come from the claims data that ACCI is using in their project, or [ Indiscernible-name unknown ] in the various and or various insurers are using. Currently the APCD data you get a full view of what is going on but then it is older and more difficult and regulations in many places and cannot do plan specific work. And we don't know what anyone's individual deductible is and so on. I don't know where I am not sure that conversation goes.

Sherry?

[ Indiscernible-speaker too distant from microphone ]

Worst and foremost I just want to say thank you to the CAHPS team for your tremendous work.

I will take that back to the team. Thank you.

We rely on you, we have pushed you and the data. Thank you for your dedication and persistence and just making it happen and making the data available so that it can be used. I do have a question though. As we move more and more towards engaging patients and families in their care, we often think dichotomous Lee, like it is the patient or their family or the patient and their surrogate, or the caregiver and the patient. I know that we have more questions directed to families or more directed to surrogate. But how ugly melded the two - but how have we melded the two? They are related but separate. Some more thoughts about that more philosophically.

For the nursing home survey we developed we have a patient questionnaire of family questionnaire, long-term and short-term, so at least in that case we have a range of surveys through which to get the information. I cannot say that we have got that on any other surveys. And we have not worked on any kind of framework to pull the information together and look at concordance or look at putting it into a total package. That is something I could bring back from the team. One issue there is a lot of times in situations where you are looking at a response, well, in the case of the nursing home there are difficulties with comprehension for different reasons. And I am not sure what is possible in terms of pulling stuff like that together into a package. I certainly think it is worth exploring and I believe they are making similar effort on the hospice survey for a patient and family, at least the patients were entering hospice at that point and how well it is meeting their needs. That is something again that I will take back to the team and we will discuss. If I get any other insights I will give you a call.

That is great. Thank you. And to build on Mary's point about the goals of care, just even knowing that the healthcare provider is aware of and knows what the goals of care are I think is a really helpful start. It gives the patient some reassurance that care is going in the right direction. But I think it is particularly germane, and thank you for bringing up the surgical outcomes, where those goals of care conversation that you have an opportunity to engage in is where those procedures are elective or not emergent. So maybe rather than thinking about a specific condition, or maybe it is a specific condition, but the context of being planned or at least elective gives you more of an opportunity to think through what your options are. So just a thought about that. So thanks.

Sandy?

I just want to let Catherine know that I really like the idea of information to help people make better choices. I think the exchanges are the beginning Affordable Care Act, I would not be surprised to see more more people going to the exchanges, particularly companies that will hit the luxury tax or Cadillac tax. I think there is a lot of thought that has to go into how to do that. It is hard to take on anything new when you're struggling to fight for what you've got right now but I think that is a real growth area for the future. And I do not think at least for now the private sector that is available is doing a good job of that. When I go to look for a doctor's contact information, somebody I know, I have to go to page 2 or three of Google or BING to get past all these grading scores or information things most of which have almost no responses and no validity. [ Indiscernible-low volume ] I think that is a really important area and it would be great for the agency if we could figure out how to do it. I am not a political expert but I do think going to Congress asking how to make the Portal Care Act or better is even I know not a winner right now. But there could be something done about the larger issue that could be done in a way that is valid and reliable. I like that a lot.

[ Indiscernible-low volume ] Picking up on Sandy's comment going back to Rick's original comment, it was not too many years ago when really we thought that no one was going to shop online. It wasn't that long ago that no one ever thought that we would trade online, that we would buy major, major products online. I am at the age right now that unfortunately I have to schedule my work around my doctors appointments. I will say that I am much more prepared when I meet my various physicians that I was before because I am using the Internet and I am of an age where just think of the young folk who have gone through this at an early age. I think they are going to be very different and I think we need to be prepared for them. I think they will be using this kind of information. And more wisely, I really do not care whether my surgeon treats me with respect, I care about my primary care physician on that.

Yes realistic expectations.

[ laughter ]

I want to know what the success rate was and when I am going to get back to work. And we will get smarter about measuring of these things. The last thing that I think we should do is walk away from this.

Any final thoughts Kathy or Chris or Ginger?

No.

Rick.

Thank you for your input, and Kathy for tracking here, and I agree Charlie, this is a good ending to the session. It is clearly really an important area and is going to be a more important area. It is still a bit of a puzzle for me given a lot of energy and money in the private sector working on this, thinking about the limited access to data that we have at the agency, and you know, figuring out the role. Kathy made a number of very clear and good suggestions about things that we might be doing and we will be working on this and coming back to you. But also if you have more thoughts on things you think we ought to be doing please do let me know.

As I understand we do not have any public comment which is part of why we were allowing ourselves the extra time. What I will say is, and I know that people are beginning to sort of pilaf, and folks should feel free to do that, but I will do this a little bit out of order. I do want to thank everybody that is around the table and has been here today for their presentations and the AHRQ staff for their support. And those that were here in person and who watched the webcast.

But what I did want to do was to give people an opportunity to make either final comments or suggest agenda topics for future meetings. We did this last time I think it was quite helpful in building the agenda that we had today. This is not your only opportunity, we take cards, letters, emails, phone calls, but if anybody from today have something that I thought they would like to see us talk about at a future meeting this is a good time to get started on that conversation. Sandy.

I wonder if it might be worthwhile bringing someone from the insurance industry in that is thoughtful and things about things this way, someone from the hospital association or somebody on the executive committee or leadership about the opportunities for collaborative work where there is a public interest and having an impartial, well-qualified group like AHRQ take the lead and figure out how to do public/partner. It might be better off for Rick to do informally or something like that, but I think it might be.

Sure. Sherry.

I was going to say something more about public/private collaboration if there is anything to talk about their.

Kevin.

There are two things, the IOM came out with her latest publication on the learning health system and you had stuff about learning health organizations, and part of that was this idea is really should be healthcare organizations that are investing in this generation and application of knowledge that has immediate benefit to them to improve their operations? It really it is applied health services research. I don't know if you have thought about that, I think he gets into this idea can AHRQ partner? I know that some of his other projects solitaire connected with organizational change and stuff like that. To me I think that is what every healthcare organization should do. And I think that with money, even if things go reasonably okay with the budget, there's never enough. For an investigator the timeliness show or slow around the classic application. It is rapid cycle and it is on the ground health services research and I am wondering how AHRQ could be more connected to that and encourage people. I don't know if people have seen the softbound book that came out for the IOM but it has CEOs with healthcare organizations talking about why this is important to them. And I have another thought as well. But they are nodding.

I think it is a very important area. Again, ideas about what we might do there would be great. One thing that we are working on in that direction is around what education and training of folks who are going to be working in these learning healthcare systems, [ Indiscernible-lost audio ].

Yes, maybe that is part of the framework. And the other one of particular interest, which comes as no surprise, I would love to hear more about the primary care portfolio and this may have been discussed it prior NACs before, but this is only my second meeting. David there is so much good stuff like of the Million Hearts but I would love to hear more of that laid out.

Let me just pick up on Kevin's comment because Rick and I were talking on break about this. The term I have heard used is little R and big R research and we wrestle with that and funders of research for the VA where increasingly there is a lot of little R research. Normally stuff that would have been domain of the researchers in terms of collecting data and analyzing it or even developing predictive analytic models is all being done in our operations shop. I am true that is - I am sure that is true and Kaiser with very sophisticated work. So that leaves the question of what is the role of the folks with the luxury of thinking five years or six months down the road? So the things we have been thinking about is understanding how data is actually used. Our partners are very good at producing dashboards and manipulating data. But in the end it really are - it has to be presented to a clinician at point of care or to a manager to highlight what they are supposed to do. Thinking about that connection of data interchange is one.

Obviously the patient end of things as a whole emerging area with patients reported outcomes. I think we have the danger that the enthusiasm of putting more and more data into the system without really thinking about what we're going to do with it is sort of the equivalent of training the patient who comes in with her notebook with everyone of the glucose measures but now it will be in their smart phone or jump right into the medical record. I do think that there are some opportunities. What can we do to help these other people who are doing stuff which 10 years ago would have been called research? Is it all around system improvement?

[ Indiscernible-low volume ]

It is a censoring function.

[ laughter ]

I have been in conversations last week with a number of different kinds of organizations that are actually all having this same issue which is an increasing number and diversity of people doing analytic work. And with I would say variable levels of sophistication and tools and not great mechanisms for triaging questions and analytic challenges to the right folks or teams. So thinking about the training needs, but also thinking about how organizations assemble and figure out how to match questions and teams with the notion that there is this continuum that I need to know in the next 15 minutes and I do not care how messy it is because it is better than me making something up all the way to we need a randomized trial that is big to figure that out. I just do not think we have a way of thinking about that. Kevin?

Part of my comment I realize was our conversation in the drive from Dulles last time Beth, this is very much central to what you're doing with your work, so this would be good. Related to that then is this whole big aided notion. Which is challenging me coming from a health services research or background, massive data, it is so much data, but the methodology and like the competition scientists, machine learning, I am just trying to use the jargon and maybe learn what it means one day.

[ laughter ]

It is a great approach to thinking about how to make sense of data and to make inferences for it. Maybe this is getting too wonky for a NAC meeting but I am curious on where you think this fits in with AHRQ, is there a paradigm shift of the agency needs to stick with, do you need to develop your own big competition all - computational data question or maybe you're already doing it.

We are not doing too much of it, we do have enormous databases but do we do a lot of machine learning? No. I guess maybe the question is for what? I don't know if it would be something that we should bring back to the NAC, maybe talk about internally and then come back.

Well not for that, is there a way to generate knowledge that will come from emerging methodologies and way of thinking that are not the classic health services research. And big data, you're thinking like HCUP or something but I think these dwarf that, even beyond claims data. There is so much stuff and if you factor in patients my chart messages and things like that, I don't know, it is something to think about maybe.

Sandy is leaving. Thanks again for the fruit.

[ Indiscernible-low volume ]

[ laughter ]

Thanks Sandy.

Rick I just want to point out that we are doing a little bit of work in that area and have been supporting the EDM Foreign which is a forum of folks using this big data related to EHRs and things like that and it is a forum and network of people trying to share stories and best practices and maybe that assuming that we could try to bring to the NAC and have discussions about additional work we could be doing in that area.

[ Indiscernible-low volume ]

Oh, no. And actually it was a ADM forum where I heard the phrase little R and big R research and they have a nice online Journal of case studies of using this.

[ Indiscernible-low volume ]

It is not a very sexy place, but AHRQ can be a useful modulator on what is a lot of hype and enthusiasm about stuff. There was a conference funded by ECRI on big data two years ago that talked about it being at the top of the gardener hype curve. And someone cited a Daniel Arieli post, the data is like teenage sex, everybody is talking about it and everyone says they are doing it.

[ laughter ]

On that note.

Sandy was supposed to be here for that.

[ laughter ]

[ Indiscernible-low volume ] Before we degrade any further, I think we should probably thank everybody. Our next meeting is November 3. Actually it is at the Humphrey building in DC. We'll have a slight change of venue, it will be like old home week for me, change of day, November 3. It will be a Tuesday and I hope you can adjust and be there. Unless there is anything else we can adjourn.

Thank you all for your energy and input. And this afternoon's discussion, but this voice discussion around appropriations was very useful and we will be sharing educational material and look forward to any comments and suggestions that you have on it.

Thank you everyone.

Election day, November 3?

[ Indiscernible-low volume ]

All right. Let me get that absentee ballot.

But it is an off year.

Yeah yeah. I do tend to vote.

[ Indiscernible-multiple speakers ] [ laughter ]

[ Event concluded ]