Passenger Environment Survey (PES) is a quality-control type survey used by a number of transit agencies to rate and monitor performance. Generally a PES would include items such as cleanliness, litter, appearance, temperature (heating and air conditioning), graffiti, trash cans, and other quality issues affecting customer perception and quality of the riding experience. There is usually a quality control criteria, and dedicated surveyors are used to gather data that summarize into periodic ratings.
In North America, the New York City Transit Authority (NYCTA), Bay Area Rapid Transit District, and Metropolitan Atlanta Rapid Transit Authority are three agencies currently using this type of surveys. However, many other transit agencies have similar surveys or quality assurance processes—but other names might be used to describe them.
Introduction
Quantitative measurements require clear and well-defined standards. The majority of PES standards were developed from the same quality control criteria used by operating departments to manage their employees. Quality control criteria are found in various agency documents. Some are codified in rules and regulations, union contracts, standard operating procedures, and other official documents; others appear in training materials, field manuals, and management procedures distributed to field staff.
Use of actual operating standards is important: it lends PES a degree of official credibility, and improves acceptance by management and the public alike. Use of a different criteria to monitor performance would be unfair to both employees and management. Additionally, use of operating standards ensures by default all field employees would be familiar with PES indicators, and thus clearly understand the yardstick by which their performance is measured.
Data Collection
Because of PES's audit and merit assessment functions, the operator must ensure data collection forces have no vested interest in survey results. Biases associated with using operating staff for data collection have been observed in many organizations. Difference between traffic checker ridership counts, conductor reported passenger counts, and ridership levels implied by fare receipts often differ in commuter railroads and other transit organizations. To provide a neutral data collection workforce, a cadre (group) of specially trained surveyors can be used. Surveyors report directly to a performance measurement unit in central administration, minimizing the influence of operating departments.
Surveyors should be hired directly from outside through the usual HR process. Therefore, they start at the operating agency with minimal pre-conceptions about operations. These procedures together ensure that PES truly represents the perspective of customers – seeing the system as they see it.
Important Features
Several features are particularly important in this type of quality control survey:
1. Quantitative Measurements are Taken Whenever Possible: Surveyors are required to count or measure quantities. Number of doors not operating is counted. Interior temperature is measured with a digital thermometer. Quantitative data enables analysis to be conducted after the fact, and reduce the risk of surveyor errors in judgment.
2. Auditing Information is Gathered Simultaneously: By collecting information such as car numbers, boarding and alighting stations and times, an analyst can later establish whether the surveyor was actually present at the correct location. The fleet numbers can be cross-checked with block register information independently recorded by operations personnel in signal towers. This is particularly critical when invariably a bad score causes operating departments to challenge the validity of one surveyor's work. If discrepancies are found, results are discarded and surveyor asked to explain how mismatching car numbers appeared in their data. Additionally, spotchecks by surveyor supervision helps to maintain data quality.
3. Failing Scores are Clearly Documented: Whenever a failing score is recorded, the surveyor records the reason on the form. This demonstrates the surveyor understands PES standards and their correct application. The data is also used to analyze root causes of failing scores. The requirement to record specifics (e.g. color of observed graffiti) allows for post-facto auditing of surveyor activity. Analysts regularly review the forms to gauge surveyors' comprehension level and act proactively to prevent recurring errors, by reinstructing surveyors as necessary.
4. Qualitative Measures Require Clearly Defined Rating Standards: Occasionally, quantitative measure is difficult or impossible to collect. An ordinal 'rating' standard is then used with clear definitions. Bottles and cans, and food matters trigger an automatic failure rating.
Sampling for PESs
Statistically valid measurements require even and representative samples. Biases due to sampling must be minimized, to ensure validity in PES's audit and merit assessment functions. At the same time, sample size must be minimized for cost reasons. Striking a balance between a statistically valid sample and minimum surveyor cost is not simple. Historical data analyses suggest PES scores are correlated with time-of-day, location, and service route, even though indicators themselves are independent. PES scores are also subject to clustering effects: data collected sequentially on the same day may be adversely affected by common factors such as a school condition, weather conditions, or special events. A pure random sample is an inefficient method of collecting such data, but observations made sequentially cannot be treated as independent observations. Generally, a good statistical sample should fulfill three criteria:
1. Sample error margins should be appropriate for the intended use of the statistic.
2. Sample should be representative of the underlying population.
3. Data collection method should not violate assumptions (e.g. assumption of independent observation) used when designing the sample.
A representative sample weighs each sub-class of the population equally. Applying that principle to PES requires analysts to determine a priori what factors might affect PES scores, and stratify the sample based on buckets of these variables. However, PES scores are affected by many factors. With few observations per reporting period and many scheduling constraints, a true stratified sample is not possible. A heuristic (rule-of-thumb) can be used to minimize the impacts of skews introduced by uneven sampling:
- Surveys should be evenly distributed throughout the whole reporting period, e.g. nine required Bus Terminal surveys are equally distributed throughout a quarter, with three surveys assigned each month.
- Where surveys are conducted at multiple locations, or roving, assigned locations and directions should be evenly distributed.
- Surveyors should be scheduled such that they will not repeat any depot, terminal, or route within a quarter.
- For surveys with a long allowable timespan (e.g. 0800-2200), samples should be evenly distributed between morning, afternoon, and evening.
Application in New York
Passenger Environment Survey at the NYCTA was introduced by David L. Gunn, former president of the Authority.
PES was introduced in the First Quarter of 1983 and consisted of a handful of very simple indicators designed to quantify how amenable and comfortable the transit environment was to our customers. Although it is difficult to objectively assess the many conditions that relate to passengers’ feelings and perceptions, the early indicators provided a systematic framework to assess performance in these areas independently of anecdotal customer comments. In that sense the early PES was a big step forward and an important recognition that NYCT cares deeply about the environment that is presented everyday to our 7.3 million customers. The early PES included 11 subway indicators, and 21 bus indicators.
Several indicators were added in 1995, expanding the scope of PES. Indicators relating to the station environment such as functioning turnstiles and escalator/elevators were added. A major revision occurred in 2000, when indicators specifically designed to measure the performance of cleaning crews were added. Quarterly reports were scaled back to semi-annual beginning in 2004. NYCT began monitoring the passenger environment aboard express buses in 2007. Additionally, the PES survey methodologies have been used on a variety of "pilot" policy initiatives designed to measure the impact of allocating additional production resources to cleaning functions. The PES approach has been such a success that NYCT regularly receives requests for details, nationally and internationally, from other transit properties wishing to model quality control programs after NYCT's PES.
Initially, PES data was tabulated manually, and reports were issued quarterly with substantial time lag. Although this enabled managers to act after the fact to improve performance, many months may pass before substantial performance improvement is achieved. With the advent of personal computers, NYCT began keeping PES data and reporting results using spreadsheet applications and databases. Internal reports are now issued much more frequently, allowing management to react to mediocre scores mid-quarter. Data mining capabilities allow special requests from top management to be handled quantitatively.
Today, NYCT routinely produces internal PES monthly reports as a snapshot to assist operating personnel in taking corrective actions. The PES database is being migrated to Transit's enterprise server. The data collection effort is being transitioned from pen and paper to handheld computers. The PES staff provides regular 'Roadshows' in field locations to improve operations managers’ understanding of PES ratings and how they might improve PES scores, thereby improving the customer experience. These PES Roadshows have become an important mechanism of information exchange, improving relationships between PES and operating staff, and allows PES staff to gain valuable understanding of operations processes.
New York City Transit's Passenger Environment Survey (PES) is a quantitative and scientific approach to measuring passenger perception of NYCT's services. The PES standards are consistent, well-defined, and clearly understood by operations personnel. The indicators are measured from a passenger perspective by a group of dedicated and impartial surveyors, and subject to rigorous data quality control by trained statistical analysts. Internally, PES functions as a performance audit for field operations management. Externally, PES serves as a dispassionate and analytical 'scorecard' measure of passenger experience.
First designed in 1983, PES has evolved over time to include 68 indicators in four distinct categories measured in four different passenger environments. Nine survey forms and 25 surveyors produce semi-annual reports with statistical precision exceeding 95% ± 5%. NYCT has such confidence in these measures that PES results are considered in the promotion and merit decision-making process. The rich and extensive dataset generated by these periodic sample measurements are used throughout NYCT to support quantitative management decision making.
References
- Ryus, P. TCRP Research Results Digest, Number 56. Summary of TCRP Report 88: A Guidebook for Developing a Transit Performance-Measurement System. Washington D.C. (2003).
- Kittelson & Associates Inc. TCRP G-6 Interim Report: A Guidebook for Developing a Transit Performance Measurement System. Washington D.C. (2002).
- "Passenger Environment Survey - Transportation Research Record: Journal of the Transportation Research Board - Volume 2112, Volume 2112 / 2009 - Transportation Research Board of the National Academies". Archived from the original on 2013-01-28. Retrieved 2012-08-15.