PASS Summit - Topics, Topics Everywhere

05 March 2015

Background

Technical conferences live and die by their community, and their content. A community conference like PASS Summit succeeds when it helps its presenters.

Several weeks ago, PASS advertised a survey, asking the community what topics and sessions they would be interested in. The survey link was sent out with the PASS Connector to 122.5K subscribers, and there were 85 responses. That's roughly typical for large email surveys.

Survey results and analysis

The raw survey results are available. However, one limitation with survey responses is sampling bias: survey results don't reflect the audience they represent because some types of people respond more than others.

For example, 10.4 % of PASS Summit 2014 attendees were consultants, and yet only 3.6% of the survey responses were. To compensate for this, I biased (weighted) the survey results to be representative of the distribution of Summit attendees. In this example, consultant responses would be weighted by 2.88 (10.4% / 3.6% = 2.88).

The weighted survey responses are on GitHub.

Overview

Session Content

Session Types

On a scale of 1-4 (4 meaning ideal interest), the most popular session content is for new SQL Server features, demos, and script examples.

Session Format

Session Formats

Tracks

Tracks

Topics of Interest

Azure

Azure Interest

Application Development

Application Development Interest

BI Information Delivery

BI Information Delivery Interest

BI Platform Architecture

BI Platform Architecture Interest

DBA

DBA Interest

Professional Development

Professional Development Interest

Interpretation

Looking at the data, I find a few interesting narratives:

  • PASS Summit is a mature conference, and many people are return attendees. As a result, many of the best practices, tips-and-tricks styles of sessions have been done before.
  • There is strong interest in panel discussions and 'birds of a feather' style events. I am guessing that's because there's not enough time to hear from all of the interesting people at PASS.
  • DBA topics haven't changed much since my SQL Server 2005 days: DMVs, performance tuning, monitoring, memory management, HA/DR.
  • BI attendees are really interested in the latest stuff: PowerPivot, self-service BI, Azure BI, and Data Mining.

There are many, many different lessons and conclusions you can draw from this data, so I'll let you speculate about the rest. Enjoy!

Permalink

Say Anything! PASS Summit Feedback and Ratings

17 December 2014

In my last blog post, I explored some of the patterns found when looking at attendance of PASS Summit 2014 sessions. Attendees also left feedback...

Note: due to PASS policy, I am not allowed to release session rating information that can identify a particular session or speaker. I have done my best to anonymize session data while retaining its full analysis information.

Feedback Loops

The way to give feedback for Summit sessions this year was using an online form, built into the PASS Summit app. People attended sessions 36,445 times and filled out 5,382 feedback surveys, for a response rate of 14.8%. That's a pretty low percentage, and I've heard that's partly because of spotty Wi-Fi and cell connectivity.

How much can we trust this data? How closely does it reflect reality?

We Don't Know

This is Statistics 101: sample sizes and populations. If we assume that the feedback is broadly typical of all attendees, then our margin of error is 1.62% (given 99% confidence).

The true margin of error is higher. The people who provide feedback are often the ones who loved a session, or hated it. Session feedback is anonymous, and without demographic and longitudinal data for each response, there's no way to know.

If I was a dyed-in-the-wool statistician, I'd stop here. Instead, I'll continue with the assumption that the data represents all attendees' opinions.

What's the Distribution of Feedback for Each Question?

Presenters get high marks.

Rankings

Session speakers are often keenly interested in their ranking. Did they get the #1 most highly-rated spot, or the #3?

Due to privacy concerns, I can't release ratings with session names or speakers. However, I can tell you the percentile rankings.

Percentile    Overall    DBA Track    BI Info Track    BI Platform Track    AppDev Track    ProfDev Track
95th 4.90 4.85 4.86 4.87 4.93 5.00
90th 4.84 4.84 4.81 4.81 4.87 4.96
80th 4.76 4.74 4.75 4.73 4.80 4.87
70th 4.71 4.70 4.70 4.67 4.73 4.80
60th 4.66 4.64 4.60 4.62 4.68 4.72
50th (Median) 4.61 4.61 4.55 4.52 4.66 4.67
40th 4.53 4.53 4.46 4.48 4.61 4.61
30th 4.47 4.39 4.40 4.44 4.58 4.56
20th 4.33 4.24 4.34 4.29 4.51 4.50
10th 4.16 4.07 4.25 4.23 4.33 4.42


Note: rankings do not include the environment scores, since that is outside of a speaker's control

Correlations

A few weeks ago I asked folks on Twitter what questions they had that could be answered from session feedback.

A few things to remember:

  • A correlation of 1 or -1 means they are 100% linked
  • A correlation of 0 means there's no relationship whatsoever
  • Correlation != Causation.

Environment Score and Speaker Performance

Is there a correlation between the "environment" score given by attendees and the speaker rating?

There's a weak correlation, (R2 = 0.377). There are also many potential reasons for this.

Enough Material and Session Length

Is there a correlation between the enough-material question and the session length?

I don't know. There's no information about which sessions ended early or late, unless you want to measure them using the online session recordings. There's not enough information when comparing regular sessions to half-day sessions to derive anything useful.

Attendance and Popularity

Cynicism and Timing

Do certain time slots produce higher scores?

There's no real correlation between time slots and scores. There is some variation of scores between times, but there's no pattern I can find to it.

Speak Up, Speak Up

Do certain times of day have higher completion rates for feedback?

Feedback is higher in the morning, but the pattern isn't consistent. There's also an outlier for the vendor-sponsored sessions that include breakfast.

The Packed-Room Effect

Does a packed room (room % full) correlate with higher or lower ratings overall? No! The correlation is very weak (R2 = 0.014), and it's not significant (p-value 0.09).

The Bandwagon Effect

Do popular sessions (total session attendance) correlate with higher scores?

Sort of. The linkage is very weak, with a correlation of 0.031 (p-value 0.012)

Consistency

Is past performance an indicator of future success? In other words, did repeat performers improve, stay the same, or get worse?

Let's compare each year's data with the speaker's overall score:

Most speakers stay within 0.5 of their average score. There are very few wild improvements or wild disappointments. However, a 0.5 difference is the difference between a 4.95 rating (amazing) and a 4.45 (in the 40th percentile).

Future Research

This work is not done. Data always leads to more interesting questions. There are many places to take this information, including:

  • Comparing speaker ratings with the scores given to abstracts to see how well session selection is doing. Let's provide data about the Speaker 47 problem.
  • Adding topic and content analysis to look for patterns by session topic
  • Investigating data-driven ways for the PASS community to build and identify upcoming speakers (cough SQLSaturdays cough)
  • Investigating how to better gather feedback during sessions. The ~14.8% feedback rate isn't good.
  • Considering letting the PASS community vote on sessions they'd like to see. This is potentially controversial, but it does have the benefit of having sessions directly represent what the (voting) PASS community want to see.
  • Using information about popularity and time slots to better allocate sessions and figure out what potential changes could be made to the schedule.
  • Telling me I'm wrong. This is great - I'd rather be told I'm making an error than let that error lead to bad decisions.
  • Providing more transparency around the session selection process.

Keep Going

There's no reason I should be the only person looking at this data. The #sqlpass community is large and technically savvy. To that end I've made almost all of the raw data public. The only piece missing is the speaker ratings for individual sessions and speakers; that has been anonymized as much as possible at PASS's request.

You can contact me anytime via Twitter (@DevNambi) or email (me@devnambi.com).

Permalink