Splash
HCUP Calculating Standard Errors
Menu
Splash
Welcome
Thank you for joining us for this Healthcare Cost and Utilization Project (HCUP) online tutorial on Calculating Standard Errors.
My name is Dave, and I am going to show you how to calculate standard errors for the national estimates calculated from the HCUP nationwide databases.
This tutorial is for researchers who have some background in basic research methods and who understand how to produce national estimates using the HCUP nationwide databases.
For a detailed description on how to produce regional and national estimates using these databases, please refer to the Producing HCUP National Estimates Tutorial.
About HCUP
Before we get started, a quick word about HCUP:
HCUP is sponsored by the Agency for Healthcare Research and Quality (AHRQ). HCUP is a family of databases, software tools, and related research products that enable research on a variety of healthcare topics. The nationwide HCUP databases are designed to facilitate the development of national and regional estimates.
If you are unfamiliar with HCUP or would like a refresher, please consider taking our HCUP Overview Course.
Learning Objectives
The goal of this tutorial is to show you how to determine the precision of the estimates you calculate from HCUP nationwide databases so that you will be able to draw sound conclusions from your analyses.
By the end of this tutorial, you will:
Understand how to calculate standard errors for the national estimates calculated from the HCUP nationwide sample databases, the Nationwide Inpatient Sample (NIS), the Nationwide Emergency Department Sample (NEDS), and the Kids' Inpatient Database (KID).
And understand how to calculate standard errors for estimates based on subgroups of the nationwide sample databases.
Importance of Calculating Standard Errors
Standard error is a measure of the precision of a statistic. It reflects the amount that a sample statistic's value would fluctuate if a large number of samples were to be drawn using the same sampling design. Less precise estimates have larger standard errors while more precise estimates have smaller standard errors.
Standard errors can be used to determine if differences between two sample statistics are significant, or construct confidence intervals for sample estimates of population statistics such as the mean.
The calculation of a standard error involves another statistical measure: standard deviation. Standard deviation measures the spread of individual data values around the mean.
For a simple random sample of size n from a large population, standard error of the mean equals the standard deviation of the sample divided by the square root of the sample size.
HCUP Nationwide Database Sample Design
The HCUP nationwide databases are not simple random samples. The NIS and NEDS are stratified two-stage cluster samples without replacement and the KID is a stratified sample. Standard formulas for a stratified two-stage cluster sample without replacement may be used to calculate standard errors in most applications for all three samples. Although a sample of hospitals is not drawn for the KID, for estimation purposes, KID hospitals are to be treated as though they were selected at the first stage of sampling from the entire universe of hospitals within each stratum. Examples provided in this tutorial use NIS data, but the same standard error calculations apply to the NEDS and KID. To review the sample designs, refer to the HCUP Sample Design Tutorial, or refer to the information below.
The Nationwide Inpatient Sample
The NIS is a stratified two-stage cluster sample. Hospitals in the sampling frame are stratified by five key characteristics. Then, a random sample of hospitals is chosen from each of the strata. In sampling terminology, each hospital is considered a cluster. The NIS includes all discharges from the selected clusters, or hospitals.
The Nationwide Emergency Department Sample
The NEDS is a stratified two-stage cluster sample. Hospital-based emergency departments in the sampling frame are stratified by five key characteristics. Then, a random sample of hospital-based emergency departments is chosen from each of the strata. In sampling terminology, each emergency department is considered a cluster. The NEDS includes all discharges from the selected clusters, or emergency departments.
The Kids Inpatient Database
The KID is comprised of a sample of pediatric discharges from all hospitals in the sampling frame. Discharges are stratified by whether they are an uncomplicated in-hospital birth, a complicated in-hospital birth, or a pediatric non-birth. For the KID, a random sample of 10% of uncomplicated in-hospital births and 80% of all other pediatric discharges is selected.
Finite Population Correction
The procedures being described today all assume inferences to a large population. Therefore, the finite population correction is not used. It is applied only when inferences are being made to the specific population of patients actually hospitalized during the year of the data. Usually analysts prefer not to use the finite population correction because they are interested in the long-run results for hospitals. For example, interest centers on the true, long-run mortality rate for a hospital rather than on the mortality rate actually observed in 2007.
For more information on instances in which it would be appropriate to use the finite population correction, please refer to the Calculating Nationwide Inpatient Sample Variances report.
Statistical Software
Several statistical programming packages can be used to calculate sample statistics and appropriate standard errors based on data from complex sampling designs. Some examples of these statistical programming packages are SAS®, SUDAAN®, STATA®, and SPSS®.
I will use SAS in today's demonstrations. In particular, I will use the SAS survey sampling and analysis procedures.
SURVEYFREQ
SURVEYLOGISTIC
SURVEYMEANS
SURVEYREG
These procedures incorporate the complex sample design of the HCUP nationwide databases into the analysis. They MUST be used when calculating national estimates, regional estimates and standard errors.
The HCUP report Calculating Nationwide Inpatient Sample Variances provides more information as well as example code for calculating standard errors using other statistical packages.
National Estimate Example
First I will show you how to produce standard errors for statistics based on the entire Nationwide Inpatient Sample. The SAS program code below produces national estimates of the sums, the means, and the standard errors for the number of discharges, the length of stay, the percentage of people who died during hospitalization, and the total hospital charges from the 2007 NIS.
In all examples, the following conventions apply:
Lowercase words denote NIS variable names.
UPPERCASE WORDS denote keywords and options that are part of the programming language as well as user-defined variables.
National Estimate Example Continued
Here is a sample of SAS code (explanations for each statement are provided below).
Standard Errors for NIS Estimates
LIBNAME NIS2007 "C:/";
DATA NIS_2007_CORE;
SET NIS2007.NIS_2007_CORE;
RETAIN DISCHGS 1;
RUN;
PROC SURVEYMEANS DATA=NIS_2007_CORE SUM STD MEAN STDERR;
WEIGHT discwt ;
CLASS died ;
CLUSTER hospid ;
STRATA NIS_stratum ;
VAR DISCHGS los died totchg ;
RUN;
SET: Keep all observations in the CORE file.
RETAIN: Create a dummy variable to ensure that every observation will be included in the discharge count.
The PROC SURVEYMEANS statement invokes the SAS procedure.
The DATA= option requests that the analysis be performed on the NIS 2007 Core file.
The SUM option requests the sum for variables listed in the VAR statement. For example, the variable DISCHGS is set to equal 1 for every record, so its sum estimates the total number of discharges.
The STD option requests the standard deviation of the sum.
The MEAN and STDERR options request that the mean and its standard error be printed.
The WEIGHT statement weights each record by the value of the variable DISCWT.
The CLASS statement identifies DIED as a categorical variable for which a ratio analysis is performed (ratio of sum of DIED to sum of DISCWT).
The CLUSTER statement specifies HOSPID as the cluster identifier. The cluster is the sampling unit—in this case, the hospital.
The STRATA statement specifies NIS_STRATUM as the stratum identifier. In the case of the NIS, the strata are based on hospital characteristics.
Example Results
Here are the results of the program.
Overall NIS Statistics
The SURVEYMEANS Procedure
Data Summary
Number of Strata 60
Number of Clusters 1044
Number of Observations 8043415
Sum of Weights 39541948
Class Level Information
Class Variable Label Levels Values
DIED Died during hospitalization 2 0 1
Statistics
Std Error
Variable Level Label Mean of Mean Sum Std Dev
dischgs 1.000000 0 39541948 799355
LOS Length of stay (cleaned) 4.588841 0.036069 181442981 3845722
TOTCHG Total charges (cleaned) 26124 561.593216 1.0135784E12 31763158175
DIED 0 Died during hospitalization 0.980633 0.000295 38767629 786809
1 Died during hospitalization 0.019367 0.000295 765651 16629
There are 60 sampling strata; 1,044 clusters, each of which is a hospital; and 8,043,415 unweighted sample records in the 2007 NIS.
According to the results, it is estimated that nationwide there were a total of 39,541,948 inpatient discharges with a standard deviation of 799,355.
The estimated average length of stay was 4.59 days with a standard error of .04 days.
The estimated average total charge was $26,124.00 with a standard error of $561.59.
The mean of the flags indicating death during hospitalization was 0.0193. In other words, 1.94 percent of stays resulted in death during hospitalization with a standard error of 0.03 percent.
Verification of Results
The results of the example analysis can be verified using HCUPnet.
When the results of the SAS program are compared to HCUPnet output, all of the estimates and standard errors agree: total discharges, length of stay, total charges, and in-hospital deaths.
Calculating Standard Errors for Subsets
What if your research focuses on only a subset of discharges from the NIS, such as hospital stays in which a coronary artery bypass graft, or CABG (pronounced "cabbage") was performed? Does calculating standard errors for a subset of discharges differ from calculating standard errors for estimates based on the entire sample?
Yes. When you produce statistics based on all the discharges in the sample, you include discharges from all of the hospitals in the sample, and thus take all of the hospitals, or clusters, in the sample into account.
If you select a subset of discharges, your subset may not include discharges from all of the hospitals in the sample.
However, to produce accurate standard errors, you must account for all of the hospitals in the sample.
The standard errors from a subset will be correct if every sample hospital has at least one observation in the subset.
There are two methods you can use to account for all of the hospitals in the sample:
1. The recommended method uses all of the records in the core file and identifies discharges of interest.
2. The alternate method subsets the database and creates "dummy" records for hospitals in every stratum to ensure the appropriate calculation of standard errors. This method is sometimes necessitated by computer memory limitations, and may be of particular use when working with the Nationwide Emergency Department Sample--which contains 27 million unweighted observations. We will look at both methods.
Subsets: Recommended Method
The recommended method for calculating standard errors requires more disk space and CPU time than the alternate method because the HCUP nationwide databases have a large number of records, all of which are involved in the recommended method. This may present a challenge in terms of disk space or software capabilities when using a database such as the 2007 NEDS--which contains 27 million unweighted observations. In this case the alternate method, which we will look at shortly, may be more appropriate. See below for an explanation of each line of code and the recommended method for calculating standard errors.
Standard Errors for Subsets: Recommended Method
/*CREATE SUBSET OF CABG PROCEDURES*/
DATA CABGSUBSET;
SET NIS.NIS_2007_CORE;
RETAIN DISCHGS 1;
CABG=0;
IF prccs1=44 THEN CABG=1;
RUN;
PROC SURVEYMEANS DATA=CABGSUBSET SUM STD MEAN STDERR;
WEIGHT discwt ;
CLASS died ;
CLUSTER hospid ;
STRATA NIS_stratum ;
VAR dischgs los died totchg ;
DOMAIN CABG;
RUN;
SET NIS.NIS_2007_CORE; Keep all observations in the CORE file.
RETAIN DISCHGS 1; Create a dummy variable to ensure that every observation will be included in the discharge count.
CABG=0; Create a variable to flag discharges for which coronary artery bypass graft, or CABG, was the principal procedure performed.
IF prccs1=44 THEN CABG=1; PRCCS1 is the data element in which the CCS principal procedure is stored and the CCS code for CABG is 44. For more information on Clinical Classification Software (CCS) and CCS codes, visit the HCUP-US CCS page.
DOMAIN CABG; Use the CABG flag in the SAS DOMAIN statement in the SURVEYMEANS procedure. The DOMAIN statement requests analyses for a subpopulation (i.e. CABG procedures) and enables appropriate calculations for statistics in each domain.
Subsets: Recommended Method Results
CABG Subset Statistics Using Recommmended Method
The SURVEYMEANS Procedure
Data Summary
Number of Strata 60
Number of Clusters 1044
Number of Observations 8043415
Sum of Weights 39541948
Class Level Information
Class Variable Label Levels Values
DIED Died during hospitalization 2 0 1
Domain Analysis: CABG
Std Error
cabg Variable Level Label Mean of Mean Sum Std Dev
0 dischgs 1.000000 0 39343279 795128
LOS Length of stay (cleaned) 4.564580 0.036223 179576868 3807345
TOTCHG Total charges (cleaned) 25690 552.813996 991755014865 31135882614
DIED 0 Died during hospitalization 0.980633 0.000296 38572861 782677
1 Died during hospitalization 0.019367 0.000296 761780 16544
1 dischgs 1.000000 0 198669 11764
LOS Length of stay (cleaned) 9.393316 0.106410 1866113 110979
TOTCHG Total charges (cleaned) 112377 3564.236638 21823434276 1344894920
DIED 0 Died during hospitalization 0.980511 0.000849 194768 11562
1 Died during hospitalization 0.019489 0.000849 3871.306347 262.549297
The data summary shows the output accounts for all 1,044 hospitals in the sample and all 8 million unweighted observations. The first set of statistics, where CABG equals zero, are for discharges which did not have a CABG performed. The second set of statistics, where CABG equals one, are for those discharges for which CABG was the principal procedure.
Results show an estimated total of 198,669 hospitalizations in which CABG is the principal procedure with a standard deviation of 11,764.
The average length of stay, indicated as L O S, is estimated at 9.39 days with a standard error of 0.11 days.
The estimated average total charges were $112,377.00 with a standard error of $3,564.24.
The mean of the flags indicating death during hospitalization was 0.0195. In other words, 1.95 percent of stays resulted in death during hospitalization with a standard error of 0.08 percent.
Verification of Results
The results of the example analysis can be verified using HCUPnet.
Subsets: Alternate Method
The alternate method for calculating appropriate standard errors is to subset the nationwide database to the observations of interest. Then, append one "dummy" observation for each of the hospitals included in the nationwide database that is not represented in the subset. The dummy observations ensure that all the hospitals in the sample are taken into account, resulting in the accurate calculation of standard error.
To do this, you must concatenate the subset of interest with the HOSPITAL file.
The Hospital File is a supplemental file which is provided with the NIS Core File. It contains a few key variables for each hospital included in the nationwide database.
Constructing this smaller database allows you to work around any memory limitations.
Including dummy observations for each of the hospitals in the NIS in the database ensures that the statistics you calculate will be accurate. The estimates produced with the alternate method are the same as those which are produced with the recommended method.
Subsets: Alternate Method Continued
Here is an explanation of the applicable lines of code and the alternate method for calculating standard errors.
Standard Errors for Subsets: Alternate Method
/*CREATE SUBSET OF CABG PROCEDURES*/
DATA CABGSUBSET;
SET NIS.NIS_2007_CORE;
RETAIN DISCHGS 1;
IF prccs1=44;
RUN;
/*CREATE ANALYSIS FILE*/
DATA NIS.CABGSUBSET;
SET CABGSUBSET
NIS.NIS_2007_HOSPITAL (IN = INHOSP
KEEP=hospid NIS_stratum);
INSUBSET = 1 ;
IF INHOSP THEN DO ;
INSUBSET = 2 ; /*ASSIGN A VALUE OUTSIDE THE SUBSET*/
discwt = 1 ; /*ASSIGN A VALID WEIGHT*/
/*ASSIGN ANALYSIS VARIABLES TO MISSING*/
DISCHGS = 0 ;
los = 0 ;
died = 0 ;
totchg = 0 ;
END;
RUN;
PROC SURVEYMEANS DATA=NIS.CABGSUBSET SUM STD MEAN STDERR;
WEIGHT discwt ;
CLASS died ;
CLUSTER hospid ;
STRATA NIS_stratum ;
VAR DISCHGS los died totchg ;
DOMAIN INSUBSET ;
RUN;
NIS.NIS_2007_HOSPITAL (IN=INHOSP KEEP=hospid NIS_stratum); Append dummy observations from the HOSPITAL file. The variable INHOSP indicates which file the observation came from. In this case, INHOSP=1 indicates that the observation came from the HOSPITAL file.
INSUBSET = 1 ; Create a flag to indicate observations that came from the CABG subset.
IF INHOSP THEN DO ; INSUBSET = 2 ; Set the value of INSUBSET to 2 to indicate the observation did not come from the CABG subset (i.e., had the CCS code = 44 for CABG procedures).
discwt = 1 ; Assign a weight of 1 to ensure the values for the observation are not weighted.
DISCHGS = 0 ; los = 0 ; died = 0 ; totchg = 0 ; Assign all values of variables of interest to 0 for non-CABGSUBSET observations to ensure that these observations will not be included in the statistical analysis.
DOMAIN INSUBSET ; The variable INSUBSET is used to indicate whether or not an observation came from the CABGSUBSET. In this case the statistics will be calculated separately for observations that came from the CABGSUBSET file and those that did not. Thus, we will only be interested in the results for INSUBSET = 1.
Subsets: Alternate Method Results
CABG Subset Statistics using Alternative Method
The SURVEYMEANS Procedure
Data Summary
Number of Strata 60
Number of Clusters 1044
Number of Observations 41376
Sum of Weights 199713.068
Class Level Information
Class Variable Label Levels Values
DIED Died during hospitalization 2 0 1
Statistics
Std Error
Variable Level Label Mean of Mean Sum Std Dev
dischgs 0.994773 0.000308 198669 11764
LOS Length of stay (cleaned) 9.344211 0.105682 1866113 110979
TOTCHG Total charges (cleaned) 111776 3539.026182 21823434276 1344894920
DIED 0 Died during hospitalization 0.980613 0.000844 195812 11562
1 Died during hospitalization 0.019387 0.000844 3871.306347 262.549297
Std Error
INSUBSET Variable Level Label Mean of Mean Sum Std Dev
1 dischgs 1.000000 0 198669 11764
LOS Length of stay (cleaned) 9.393316 0.106410 1866113 110979
TOTCHG Total charges (cleaned) 112377 3564.236638 21823434276 1344894920
DIED 0 Died during hospitalization 0.980511 0.000849 194768 11562
1 Died during hospitalization 0.019489 0.000849 3871.306347 262.549297
2 dischgs 0 0 0 0
LOS Length of stay (cleaned) 0 0 0 0
TOTCHG Total charges (cleaned) 0 0 0 0
DIED 0 Died during hospitalization 1.000000 0 1044.000000 0
1 Died during hospitalization - - - -
The alternate method produces the same correct statistical output as the recommended method. Again, results of the analysis can be verified using HCUPnet.
Verification of Results
Remember, if the alternate method was not correctly applied, and all hospitals in the sample were not included in the analysis, standard errors will be incorrect.
Here is an example of output from a program which does not account for all hospitals in the sample. The number of strata and clusters do not reflect the complete sample. The standard errors produced when all hospitals are not accounted for are incorrect and will lead to erroneous conclusions in your research. It is critical to ensure you obtain a correct standard error.
NIS CABG Subset Statistics Without Appending Dummy Observations
The SURVEYMEANS Procedure
Data Summary
Number of Strata 34
Number of Clusters 239
Number of Observations 40332
Sum of Weights 198669.068
Class Level Information
Class Variable Label Levels Values
DIED Died during hospitalization 2 0 1
Statistics
Std Error
Variable Level Label Mean of Mean Sum Std Dev
dischgs 1.000000 0 198669 9289.821822
LOS Length of stay (cleaned) 9.393316 0.101364 1866113 87869
TOTCHG Total charges (cleaned) 112377 3462.359115 21823434276 1097229239
DIED 0 Died during hospitalization 0.980511 0.000839 194768 9142.090456
1 Died during hospitalization 0.019489 0.000839 3871.306347 221.637623
Using the Z-Test Calculator
Once you have calculated standard errors for the subset of discharges you are studying, you may want to check to see if there are any statistically significant differences between outcomes or measures of hospital stays in your subset and those in other subsets.
The Z-test calculator is convenient way to do just that.
It can be accessed by clicking the Z-test calculator link below any HCUPnet query results page.
The Z-test calculator allows you to test the significance of the difference between two weighted counts, means, or percentages.
Z-Test Calculator LOS
To test if the length of stay of a discharge with a principal procedure of CABG is significantly different from that of stays which did not have a principal CABG procedure, enter the estimated length of stay and corresponding standard error for the CABG discharges, and then enter the estimated length of stay and corresponding standard error for discharges for which CABG was not the principal procedure. Then select calculate. The calculator provides the associated standard error, z statistic, and p-value for the test.
Z-Test Calculator Trend
To test if the number of discharges with a principal procedure of CABG in 2007 is significantly different from that in 1997, enter the 1997 estimate and corresponding standard error and then the 2007 estimate and corresponding standard error. Select calculate. The calculator provides the associated standard error, z statistic and p-value for the test.
Key Points
As you calculate sample statistics and standard errors from the HCUP nationwide databases, you should consider the following key points:
Resources and Other Training
If you are looking for more information on the subject matter covered here, several resources are available on the HCUP User Support website: www.hcup-us.ahrq.gov.
If you can't find what you need, feel free to email the HCUP Technical Assistance staff at hcup@ahrq.gov. AHRQ has senior research personnel available to respond to technical questions you may have. Inquiries are answered within three business days.
Thank you for accessing this module. There are several other HCUP Online Tutorials. Take a look to see if there are other topics that could be helpful to you.
If you have any feedback regarding this module, please email us at hcup@ahrq.gov.