ESDS Government

The large-scale government surveys are large datasets collected either by the Office for National Statistics or other agencies, such as the National Centre for Social Research. The surveys are principally designed for government information purposes but are available to researchers for secondary analysis. They are key data sources for social scientists and include surveys such as the Labour Force Survey, General Household Survey, Family Resources Survey and Expenditure and Food Survey. A full list of the surveys covered by the ESDS Government Data Service is available from the survey pages of the ESDS Government web site. You can find full descriptions of the surveys, together with documentation, on the web site - you will need to search for the survey on the homepage or follow the link from Major Studies to the survey you are interested in. There are other large-scale government surveys which are not covered by ESDS Government but are available from the general ESDS service.

2. Who can obtain survey data?

Researchers, students and teachers from all fields and types of organisation may register with the Economic and Social Data Service (ESDS) and obtain survey data. However, some datasets may have restrictions on access. For example, commercial usage may be restricted, permission may be required from the depositor or publications may need to be vetted by the sponsoring organisation.

3. How much will it cost?

All users requiring data for non-commercial purposes can download data free of charge. Where data is required for commercial purposes there is a fee - see Charges information

4. How do I gain access to the survey data or explore it online?

To access survey data, all users must register with the Economic and Social Data Service (ESDS). Information about obtaining data from the ESDS can be found on Accessing data page.

Registered users can download/order a large and increasing number of datasets direct from the web site (usually in SPSS, STATA or tab-delimited formats) via its online catalogue record and via the download/order section of the Major Studies web pages.

An increasing number of datasets are also available to most registered users in the Nesstar system, which will allow you to explore the data online and to do basic exploratory analysis before choosing to download all, or a subset, of the data.

5. What formats are the data available in?

Most datasets are available in SPSS, STATA and tab delimited formats (suitable for use in Excel) and can also be converted to other formats such as SAS. Other systems, including Nesstar, may also provide additional formats, such as Statistica, DIF (suitable for use in Excel), NSDStat and Dbase.

6. Are there restrictions on the use of the survey data?

Restrictions on the use of the surveys are outlined in the End User Licence that all users sign when registering for an ESDS account. In particular there is a fundamental restriction on the confidentiality of data. Users should not attempt to use the data to deliberately compromise the confidentiality of individuals or households and are required to abide by the current Data Protection Act. This agreement also covers requirements for citation of publications and safeguarding of data. Sharing Data with colleagues, other researchers or students must always be authorised by ESDS. Certain datasets may also require depositor permission. Some datasets which pose a higher risk of disclosure require a Special Licence (SL). This licence has additional conditions. For example, the SL requires the signature(s) of the researcher(s) and the institution with responsibility for the researcher. It also needs the explicit permission of the data owner to release the data to the researcher(s). See FAQ 29 for more detail on special licences.

7. What information is available in the surveys?

A wide variety of information is available from the surveys. Most surveys cover a specific topic, for example the Labour Force Survey (LFS) collects detailed data on economic activity and the Health Survey for England (HSE) collects information about the nation’s health and the number of people with specified health conditions among other things. In addition, each survey asks a number of core questions such as sex, age, ethnic group, housing tenure. The ESDS Government and Survey Question Bank web sites give full descriptions of the surveys. There is also work on the harmonisation of concepts and questions by ONS and on derived variables over time by ESDS Government.

8. Can I order publications and obtain statistics?

The Economic and Social Data Service (ESDS) is not able to supply copies of publications (other than User Guides accompanying the data). However, the ESDS Government web pages contain a searchable publications database which cite the ESDS Government surveys. You can search on keywords such as 'employment' and can specify whether you want to search on all the supported surveys or specific surveys. Results can be ordered by date or author. In addition, within each specific survey page the green box titled ‘links’ generally provides links to recent reports for that survey.

ESDS does not typically supply tailor-made statistics, although it will undertake certain consultancies at cost. A number of datasets are also available to most registered users to analyse, subset and download online via the Nesstar service, where basic frequency counts are freely available to all users.

9. What can I expect of the ESDS Government Data Service?

ESDS Government, led by CCSR , aims to exchange information between users and producers, enlarge the user base and provide user support. This includes:

a helpdesk service operated by email (govsurveys@esds.ac.uk) and telephone +44 (0)161 275 1980

a JISCmail list for parties interested in the large government surveys, for discussion and to receive updates. To join the list, you simply need to either go to the ESDS-govsurveys JISCmail web site or alternatively you can send an email to listserv@jiscmail.ac.uk with the following text: subscribe esds-govsurveys firstname secondname

web-based Frequently Asked Questions

a twice-yearly newsletter

dedicated survey pages

products that provide easy routes through the complex documentation of the government surveys such as introductory guides and indexes of common variables

workshops, conferences and courses on: the government datasets and their potential; key topics of interest; specific statistical packages, particularly SPSS and STATA; methods of statistical analysis; some of the unique aspects of the datasets; topic-related online course materials

a range of teaching datasets

a searchable database of publications which cite the government surveys

Further information, including the list of datasets supported by the service, can be found on the ESDS web site .

10. Who can use ESDS Government services?

ESDS Government is funded by the ESRC and the Joint Information Systems Committee to support UK academic users of the large-scale government surveys. However, non-academic users and overseas academics have access to all our materials, such as the SPSS guide, the newsletter, all other online documents and are very welcome to attend the workshops and events we organise. ESDS Government will also provide quick answers to helpdesk queries from non-academics and overseas academics. However, lengthy helpdesk queries requiring one-to-one attention can only be provided to UK academics.

11. How do I get help if I am stuck?

You can email us at govsurveys@esds.ac.uk or telephone us on +44 (0)161-275-1980.

12. Where can I get the large-scale government survey questionnaires from?

Questionnaires and other survey documentation can be found on the web site (you do not have to be registered to use this facility). The Survey Question Bank based at the University of Surrey also provides free online questionnaire and related material, e.g. diaries, for most of the large-scale government surveys from 1991 onwards. Often questionnaires can be found in the back of published reports. If you cannot find the questionnaire you require email us at govsurveys@esds.ac.uk or telephone us on +44 (0)161-275-1980.

13. What are missing values?

Missing values are categories in variables for which there are not valid responses. Missing values are specified for a number of reasons and usually have specific codes within the large-scale government surveys. Data can be specified as missing for the following reasons:

the question does not apply to the respondent (DNA). For example someone who said that they were a non-smoker would not be asked the question ‘How many cigarettes do you smoke on average per week?’. DNA is often coded –9 or 99

the respondent has simply not given an answer (NA). This is often coded –8 or 98

other reasons such as proxy response, refused entire section and so on. These are often coded as other values such as –6 and –7

you should check the survey documentation rather than assuming that the codes given above are correct.

14. Should I include missing values in my analysis?

You should never include missing values coded as DNA (see FAQ 12 above) because the question is not relevant to these cases. You may or may not want to include other missing values such as NA (see FAQ 12 above). Care should be taken when excluding missing values if they account for a large proportion of the answers because it could lead to the misinterpretation of results. The fact that a large proportion of respondents did not answer the question may be an interesting finding in itself. For example:

Have you ever used an illicit drug?
Response	N	Missing answers included (%)	Missing answers excluded (%)
Yes	50	10%	25%
No	150	30%	75%
NA	300	60%	-

If missing answers are included then you report that 60% of respondents chose not to answer this question. However, if missing answers are excluded you report that 75% of respondents have never used an illicit drug. These two findings have very different implications.

15. Why am I not getting the same numbers in my analysis as the published results?

You may not always get exactly the same numbers as the published results because of small things you may have done differently in your data preparation. However, one would expect your results to be very similar to the published results. If your findings are significantly different there could be a number of explanations:

• you may not be comparing the same variables
• you may not have dealt with missing values in the same way as the published results
• there may be a filter variable to be applied (a filter variable is used where analysis is performed on a subset of the data, for example, the analysis may have been run selected ages, such as the working aged population only).
• there may be a weighting variable to be applied.

16. What is the ILO definition of unemployment?

Most of the large-scale government surveys use definitions of employment, unemployment and economic activity and inactivity that have been internationally agreed by the ILO (International Labour Organisation). To be unemployed, according to the ILO definition you must be: not in employment but available to start within two weeks, and have either looked for work in the last 4 weeks or waiting to start a new job. ILO employment includes those who are in paid work either as an employee or self-employed and those on a government-supported training scheme. For more information see ONS harmonisation web site .

17. Which years were missed from the GHS?

The General Household Survey has been carried out annually since 1971, with the exception of two years: 1997 (when it was being reviewed) and 1999 (when it was being re-developed). In 2005 the GHS(L) was introduced – see FAQ 30 for more information.

18. Why do we weight data?

It is common for social surveys to use some form of weighting; this is for a number of reasons:

• to correct for bias caused by particular groups in the population not responding to the survey. This is called non-response weighting and often uses figures from the census to provide a correct population distribution, for example in the 2001 General Household Survey

• to remove bias caused by the use of different selection probabilities. For example, in surveys where only one adult per household is interviewed, those living in households with more than one adult will have a lower probability (or less of a chance) of being selected than those adults living on their own. The 2000 British Crime Survey and the 2001 British Social Attitudes Survey both use this type of weighting

• to make sure that the proportions of people in each age group, sex and region, for example, are the same in the sample as they are in the whole population. This is called calibration weighting (or population based weighting)

• to gross the survey to population totals, so you can say how many people in the UK watch TV every day for example, rather than just what proportion (or %). This is called grossing and is used in the Labour Force Survey.

In all such cases, the samples do not accurately reflect the general population. The purpose of weighting is to adjust the sample so that the sample profile on key variables reflects that of the population. Weighting involves statistically increasing or decreasing the numbers of cases with particular characteristics so that the proportion of cases in the sample is adjusted to the population proportion. For more detailed information on weighting please refer to the Weighting the Social Surveys guide on the ESDS Government web site .

19. Why are response rates decreasing on surveys?

Non-contact with the sampled respondent and refusal to participate are the two reasons why response rates have fallen across surveys in recent years. Tactics used by survey organisations, such as calling more frequently and at times of the day when people are more likely to be at home, help to reduce non-contact rates. However, the increase in refusal to participate is more difficult. Much of the increase in refusals seems to be attributable to increased reluctance among members of the public to take part in surveys. These general attitudes are hard to change. Instead survey organisations concentrate on the aspects they have some control over, namely how they approach individuals, how interviewers introduce themselves on the doorstep, and how individuals might be persuaded to participate.

20. Who are the non-responders in surveys?

In practice, relatively little is known about the characteristics of non-responders. On most surveys the characteristics of non-respondents are ‘inferred’ by comparing the survey profile with the population profile derived from official statistics such as the Census. But, in other than census years, this tends to restrict us to comparisons of the age and sex profile. These comparisons typically demonstrate that young men are disproportionately missing from surveys; what they don’t show is whether the missing young men have different characteristics to the responding young men. Other studies have shown that surveys typically under-represent those with lower educational qualifications, adults without dependent children, and those living in London. Knowing about the characteristics of non-respondents allows survey researchers to reduce bias by applying non-response weights to the data. See FAQ 17 for more details on non-response weights.

21. How do I use SPSS and STATA?

ESDS Government has written introductory guides to using SPSS and STATA. These can be found on the resources page of the ESDS web site.

22. Are there are training courses on how to use SPSS and STATA?

The Cathie Marsh Centre for Census and Survey Research (CCSR) at the University of Manchester regularly hold short course on SPSS and STATA at varying levels (introduction, advanced etc). The CCSR web site holds the information on the short courses, including how to apply for a bursary.

23. What is the Continuous Population Survey?

The Continuous Population Survey (CPS) is now known as the Integrated Household Survey (IHS). See FAQ 26 for details.

24. Can I use the data for teaching?

The datasets can be useful resources for teaching. You can use the datasets for hands-on exercises in methods courses such as questionnaire design or statistics ESDS Government have created a number of teaching datasets for this purpose using the Labour Force Survey, British Crime Survey, Health Survey for England and the General Household Survey (social capital teaching set). If you wish to use one of the teaching datasets or any of the ESDS Government datasets with your students as part of a taught course you should read the information on Sharing Data on the ESDS web site.

You could also use the datasets as research design examples in methods lectures or alternatively you may wish to use the data for substantive courses, for example if you wanted to show that certain ethnic minorities have high unemployment rates you could use the data yourself to illustrate this or use existing exemplars from published research.

25. I want to carry out a time-series analysis but some of the variables have changed over time. Are there any variables that are consistent over time?

ESDS Government has carried out some work to document information about the consistency of key variables over time on the General Household Survey and Labour Force Survey, and to provide syntax to derive consistent variables. The initial work is available from the ESDS Government web site and other variables will be added from time to time. Additionally, a policy of harmonisation since the early 1990s has greatly increased the consistency of definitions and question wording across key variables (but not all variables) on the main government surveys - for more information see the ONS harmonisation web site.

26. What is the Integrated Household Survey? (formerly Continuous Population Survey)

Fieldwork for the Integrated Household Survey (IHS) (formerly the Continuous Population Survey (CPS)) is proposed to begin in January 2008. The design involves the integration of the household surveys on which the Office for National Statistics (ONS) leads into one IHS (CPS). The surveys included for integration are:

• Labour Force Survey (LFS) and associated boosts (APS)
• General Household Survey (GHS)
• Expenditure and Food Survey (EFS)
• Omnibus Survey (OMN)

Two consultation meetings on the IHS (CPS) took place in January 2006 and March 2007 at the Royal Statistical Society, London. These meetings provided opportunities to find out the latest developments from ONS. They also provided opportunities to ensure that users needs were heard and considered. The content of the core outputs and each interview combination were discussed among other things. Slides from the meetings are available on the Past Events web pages. More information about the IHS is available from the IHS page on the ESDS Government web site.

27. How can I find out who has been researching using a specific survey?

To find out who has recently registered to use a survey, the first place to check is the ‘Registered Uses’ box within each specific survey page on the ESDS Government web site. When registering with ESDS for the use of data there is an option to make registration of that usage, including details of any project surrounding it, widely available.

It may also be useful to do a search on the specific survey name in the ESDS Government publications database.

Alternatively, if you have a particular research area/question and are interested in making contact with other researchers it may be useful to join a JISC mailing list, such as the ESDS Government JISCmail list. Other than the ESDS Government list, there are also the Labour Market Statistics User Group JISCmail list, Health Statistics User Group JISCmail List and British Crime Survey JISCmail list.

User meetings on a specific survey or research topic can also be a helpful way to find out current research examples, see ESDS Government events for forthcoming user survey/topic meetings and past events for presentation sides from previous events. For more information on user meetings see FAQ 28.

28. What are the ESDS Government user meetings and are they for me?

ESDS Government organises a number of user meetings each year, these are:

• General Household Survey User Meeting
• Labour Force Survey User Meeting
• Health Surveys User Meeting

• Crime Surveys User Meeting

• Family Resources Survey User Meeting

These are days which allow users to meet data producers. The days take a fairly standard format in which the morning consists of presentations by the data producers and provides information on news and developments on the surveys, while the afternoon is led by users in researchers describe new or recent work being done with the data. The meetings are friendly, informal and open to all users or potential users of the surveys. Previous programmes and slides are available from the past events section on the ESDS Government web site.

These meetings are part of a wider range of events organised by ESDS Government, see the events web page for more information. If you would like to be kept informed of all the latest events, please subscribe to the ESDS Government JISCmail list.

29. What is a special licence and special licence data?

When you require access to social survey microdata, for example, the data held at the UK Data Archive (UKDA), this requires registration, agreement to the End User Licence (EUL) and the registration of usage details.

A Special Licence (SL) is used for special licence data; this is data which pose a higher risk of disclosure meaning the licence has additional conditions. For example, the SL requires the signature(s) of the researcher(s) and the institution with responsibility for the researcher. It also needs the explicit permission of the data owner to release the data to the researcher(s).

SL applicants are screened by UKDA and the data depositors. Data are only provided to those researchers who can justify their need for the data for statistical research purposes and agree to meet the terms of the SL.

The first ONS social surveys to trial this new access initiative are the Annual Population Survey (APS) and the QLFS (from March-May 2005), but it is planned to extend the special licences to other surveys in due course.

The QLFS special licence version contains the same content as the QLFS end user file but has additional geographical information at local authority level. A list of the variables available in each file is available in an Excel file on the ESDS Government web site.

The source for this FAQ originates from Special Licence: new arrangements for access to social survey microdata on the ESDS web site.

30. What is the new GHS(L)?

The 2005-2006 GHS fieldwork is the first to be undertaken under a new longitudinal survey design. The new design is also accompanied by a slight change in substantive emphasis, which leans towards a greater range of questions on social exclusion. The changes to the GHS aim to satisfy new EU requirements to produce comparable data from a Survey of Income and Living Conditions (EU-SILC). For more information go to the slides from the 2006 GHS User Meeting.

31. When did the QLFS shift from seasonal quarters to calendar quarters?

In accordance with EU regulations, the LFS moved from seasonal (spring, summer, autumn, winter) quarters to calendar quarters (January-March, April-June, July-September, October-December) in 2006. The last seasonal quarter dataset issued was the Quarterly Labour Force Survey, December 2005 - February 2006 (SN 5356) and the first calendar quarter dataset was the Quarterly Labour Force Survey, January - March, 2006 (SN 5369). Users should note that there is some overlap between these two datasets. ONS have produced a limited series of historical LFS datasets on a calendar-quarterly basis. This will allow users to make meaningful comparisons of labour market statistics from the LFS microdata over time. Further information on the seasonal to calendar quarter change and its impact on LFS data may be found in the following online article:

Madouros, V. (2006) Impact of the LFS switch from seasonal to calendar quarters: an overview of the switch of the LFS to calendar quarters and the potential effects of this change on users, London: ONS.

32. How do I use the weights in the 2004 ethnic boost dataset if combining the general population and ethnic boost files?

ESDS Government have written a detailed briefing note on how to use the 2004 Health Survey for England (HSE) weights.

33. Can I use multiple imputation to deal with missing data in the Health Survey for England?

Multiple Imputation (MI) for missing data in the HSE can be carried out using the ice and mim commands in STATA. The results from the MI can then be compared with the results before MI is applied.

However, the ice and mim commands in STATA cannot be carried out in conjunction with the STATA svyset commands (the svyset commands adjust the analyses to take account of the complex survey design). As the HSE has a complex survey design it is important that you use the svyset commands for analyses. Using the svyset commands is likely to have a larger effect upon the results of analyses than using MI. For more information go to the Missing Data web site.

34. How do I use the Health Survey for England individual weights if I’m pooling the data from the 1999 and 2004 datasets (ethnic boost years)?

The 1999 and 2004 Health Surveys for England (HSE) both contained ethnic boost samples in order to increase the number of ethnic minority respondents in the survey.

It is possible to pool the 1999 and 2004 datasets to further increase the number of ethnic minority respondents for your analyses. When pooling the 2 datasets you need to ensure that you use the weights from 1999 and 2004 in the correct way.

First you need to prepare the 2004 weights, see the detailed briefing note on how to use the 2004 Health Survey for England (HSE) weights.

Next, you need to prepare the 1999 weights which are less complex than the 2004 weights.

For the 1999 dataset you need to follow the same procedure as 2004 in terms of dropping the ethnic minority cases from the general population dataset, merging the 2 files together and adjusting the weight so that the weighted total equals the actual total. All of this is illustrated in the detailed briefing note on how to use the 2004 Health Survey for England (HSE) weights. You can check the population is in the correct proportion for 1999 (e.g. the white group reflects the correct proportion of the population) by comparing with the 2001 Census estimates.

You should then scale the weights for both years (i.e. divide each year by the mean weight). Scaling the weights for both years means you're not giving prominence to any one year - both years have a mean weight of one. You then simply give the new scaled weight in each year the same name (e.g. wt_intnew) and then combine the datasets and use the new scaled weight (e.g. wt_intnew) as described in the detailed briefing note on how to use the 2004 Health Survey for England (HSE) weights.

ESDS Government has also written some general guidance notes for users who want to pool the 1999 and 2004 HSE datasets.

35. Where do I get statistics on...?

Unfortunately ESDS Government do not have the resources to produce bespoke analyses. Our role is to support users who are undertaking analysis of major government survey series held at the UK Data Archive.

No software?
The Nesstar system contains many datasets in a user friendly online format. Before you are registered you can browse documentation including frequencies of variables. Once registered, you can create tables and graphs for all or a subset of cases and weight these.

Do you want published statistics?
There are an increasing number of web sites providing access to statistics on the sort of social topics covered by ESDS Government. These include:

Office for National Statistics
Access to statistics for particular neighbourhoods
Open data web site for the UK Government, run by the Cabinet Office and including administrative statistics
Nomis, established by ONS to provide up-to-date tabular data on labour market statistics, including those for sub-regional geography
MIMAS, provides census tables down to small geographies
Home Office, for general crime statistics pages
Home Office, for the British Crime Survey published statistics

Additionally, many surveys result in published reports. These results are typically available in the ESDS Government resources section for the survey series in question. So, for example, you can find links to the published reports of the General Household Survey on the GHS resources web page.

Links to the published reports of the Health Survey for England can be found on the HSE resources web page.

Updates

The FAQs are regularly updated; the latest version is always available in HTML from the ESDS Government web pages .

This version was last modified on 5 March 2010

ESDS Government home

Latest News

British Secondary Data: Researching Housing and the Local Environment

Ethnicity guide updated

Opinions and Lifestyle Survey user meeting

Living Costs and Food Survey user meeting

Guide to the LCF/EFS updated

Lifestyles Surveys: Consultation Review

More news

Upcoming Events

Recent Data Releases

Recent Publications

ESDS Home Page > Government > Faq
		Page last updated 19 September 2011 © Copyright 2003 - 2012 Universities of Essex and Manchester. All rights reserved. Contact \| Copyright and Disclaimer \| Accessibility \| Send us comments on this page