Statistical Analysis Software Trends
Statistical analysis software is designed to apply the power of statistics to your company’s data. Applying statistical techniques to your data accomplishes two equally important tasks.
First, statistical analysis can provide empirical answers to questions about your business, such as “which of our two marketing campaigns resulted in more sales?”
Second, you can express your uncertainty about your estimates by expressing the level of statistical significance, or the size of a confidence interval or a margin of error.
Even with complex data and complex business questions, the right statistical analysis software will give you the tools you need to come to an answer, and express your level of certainty.
Why use statistical analysis software?
Many business decisions are made by gut instinct. The rise of data-driven business, from “Moneyball” to high-powered financial trading, has demonstrated that empirically-based decision making can result in far superior results.
However, just looking at raw numbers has its limitations. If you have large datasets, complicated analysis questions, or if you want to be able to express your level of uncertainty, you need to perform a statistical analysis.
Statistics deals with the science of uncertainty, and even for relatively straightforward questions, like whether one marketing campaign produced more revenue than another, there are several computational details that are difficult to implement on your own.
If you have a more complex analysis question, like how purchasing patterns among your customers vary as a function of age, gender, geographic location, and occupation, you can completely forget about doing that analysis from scratch. The only viable option is using a statistical analysis software package.
Using statistical analysis software can help prevent you from being fooled by random fluctuations in your data. Testing for statistical significance can give you empirical results to support or refute conclusions you draw from your data. For example, even if you ran two identical marketing campaigns, the revenue they generate is likely to differ, just by random chance.
Statistical analysis software allows you to put a number on these fluctuations, and test whether the results you actually got are likely to have occurred due to random chance.
If you were making business decisions based on random fluctuations, when no true underlying relationship actually exists in your data, you’re no better off than trusting your gut instincts. Statistical analysis software can help prevent this problem, and gives you proper data on the uncertainty inherent in your data analysis.
Statistical analysis software can allow you to build predictive models. Suppose you wanted to build a prediction model to estimate whether or not a new sales lead will ultimately make a purchase. Using statistical analysis software on your historical data, you can build a prediction model that incorporates data about the sales lead, like location, purchase history, products they are interested in, and so on.
Once you’ve built your predictive model and tested its accuracy on your historical data, you can use it to predict the likelihood that a new sales lead makes a purchase. Lead scoring in CRM software and lead generation software is based on this exact principle: using your own in-house data and your statistical analysis software, you can build a predictive model and apply it to brand-new data.
Another advantage of statistical analysis software is that it can tell you which of your predictive variables are statistically significant predictors of converting a lead into a sale.
This process of statistical inference can help isolate the most important factors that determine success, which is one of the distinctive advantages of statistical analysis over basic data visualization and data summaries.
Statistical analysis software can allow you to conduct A/B testing. Frequently in this article, we’ve used the example of a company running two different marketing campaigns, and wanting to know which was more successful.
The process of A/B testing is a formalization of this exact problem. The idea for A/B testing comes from medical research, and works as follows. First, you take a random subsample of your customers, sales leads, or email newsletter subscribers, and randomly allocate them to one of two groups.
Then, each group is started on a different marketing campaign: campaign A and campaign B (hence the name A/B testing). After each of these campaigns are over, you can compare the effectiveness of each campaign, as measured by click-through rate, conversion rate, or some other performance metric.
Statistical analysis software is necessary to analyze the results of A/B testing, since you’ll always get some random fluctuations between your two groups. A/B testing is a great tool for testing out different marketing campaigns, sales promotions, incentive structures, and more.
Who uses statistical analysis software?
Applying statistical analysis correctly requires some level of training, so the most frequent users of your company’s statistical analysis software will be people with a good bit of quantitative data analysis skills:
Data analyst. Data analysts will be the most frequent day-to-day users of your statistical analysis software, because they are often tasked with answering basic questions from your company’s data, like which factory is producing the most products over the past year, what the distribution of your customer demographics looks like, and so on.
Data analysts will spend less time on advanced statistical techniques, so for them, high-end features are less important than ease of use. For your data analysts to be as efficient as possible, you’ll want a statistical analysis software that is fast and simple.
Data engineer. Data engineers spend most of their time getting your company’s data ready for use in statistical analysis software or data visualization software.
While they won’t do much in the way of performing complicated statistical analyses themselves, your data engineers will still be using your statistical analysis software to load in data, see if there are any problems with it, and check for outliers and missing data. They’ll be particularly interested in easy-to-use tools for finding problems with your data.
Data scientist. Data scientists will be the users who are most interested in advanced features like clustering, machine learning, and nonparametric regression. Data scientists distinguish themselves from data analysts by exploring new, unanswered questions, and discovering surprising or novel patterns in your company’s data.
Data scientists will likely already have strong programming experience and a deep understanding of the math behind statistical analysis tools, so features like a graphical user interface will be less important for them. Instead, they’ll prefer software that supports using scripts for reproducible and scalable analysis, and software that combines statistical analysis with data visualization.
Actuary. Actuaries are specifically concerned with quantifying risk, so they’ll be using your company’s data to predict probabilities of accidents, loan defaults, and other high-risk scenarios.
Actuarial science has its own set of statistical tools that shares some overlap with normal statistical techniques in broad use, but you may want to talk to your actuaries to see if they have specific techniques, like survival analysis, that they will need to be able to perform in your statistical analysis software.
Research scientist. Your research and development team will use a broad range of statistical tools tailored to the specific problems they’re working on.
Research scientists will be particularly interested in statistical analysis software that makes it easy to run the same analysis many times in a row, possibly with small modifications, as they develop and refine their research project.
Research scientists also need basic data visualization capabilities, but not to the same extent as a data scientist, since their data visualizations will be mostly exploratory and for internal use.
Features
The biggest point of difference among all statistical analysis software packages is whether they use a graphical user interface or a scripting and command line interface. Statistical software comes down on one side or another when it comes to graphical user interfaces, or GUIs.
Either the software is based around menus and buttons, or it’s based around scripts and command lines. Most software you are familiar with, like Microsoft Word or Powerpoint, is GUI-based. Command lines and scripts have their roots in programming languages (and indeed, many script-based statistical analysis software packages are programming languages).
GUIs are much easier to use and to learn, but ultimately scripts and command lines are far more powerful, far more flexible, and lend themselves better to reproducible analyses. If the users of your statistical analysis software will mostly be data analysts, you’ll want to lean towards using a GUI-based software like SPSS, JPM, or Prism GraphPad.
If your data scientists and research scientists will be the main users, lean toward scripting software like R, Python, SAS, or MATLAB.
Strong data visualization capabilities can extend the use of your statistical software package. Normally it can be risky to try to get software that accomplishes two things at once, since you risk ending up with a product that does both of them mediocrely well, and neither great.
However, since statistics and data visualization are so closely intertwined, you may want to put more weight into considering the data visualization capabilities of your statistical analysis software.
All statistical packages will produce some kinds of plots, but higher-end software (and script-based software) can usually produce better plots. This can be one of the defining features of better statistical analysis software.
Make sure your statistical software is well-documented. Since it’s hard to tell up front all of the kinds of statistical analyses you will want to run in your statistical analysis software, your employees who use the software will inevitably end up having to look up information in the software’s online documentation frequently after you get it.
Good documentation can make carrying out a complex statistical analysis much easier, and the converse is true for sparse or poorly-written documentation.
Popular statistical analysis software tends to be easier to troubleshoot. With statistical analysis software, one big reason to gravitate towards larger and more popular statistical software packages is the fact that they have much more in the way of online resources for help, both formal and informal.
For software like SAS, SPSS, MATLAB, R, or Python, a quick internet search can find a solution to just about any bug, technical glitch, or problem you are having. Troubleshooting niche software can be trickier, because the user base is so much smaller.
Some statistical analysis software requires purchasing additional add-on libraries for advanced functionality. Depending on the manufacturer, your statistical software may be one monolithic program, or it may rely on many different libraries and tools.
R, for example, relies heavily on libraries that are freely available on an online repository. Other software, like MATLAB, requires you to pay extra for advanced features like machine learning or real-time data streaming.
If you know that you will have specialized data analysis needs, make sure you check to see if all of the statistical functions you want are included in the base capabilities of your statistical analysis software, or if you will have to pay extra for the advanced functions that you need.
If you work with messy data, be sure to look into the data cleaning capabilities of your statistical analysis software. Data scientists and data analysts spend a huge amount of time “cleaning” data, meaning organizing it, formatting it, and preparing it for formal statistical analysis for visualization.
With modern statistical analysis software, running a statistical analysis is often the fastest and easiest part of the whole project, once you know what analysis technique you’ll be using.
If your data scientists and data analysts have to clean a lot of data, or if you use large, real-world datasets with a lot of missing values and potential outliers, you should definitely investigate what types of outlier detection and missing value imputations tools are available in your statistical analysis software.
FAQ
Q: Are there any free statistical analysis software packages?
A: Yes, unusually for business software, some of the most powerful statistical tools are also the ones that are completely free. R and Python, the two languages that are most-used by data scientists and researchers around the world, are both free and open-source.
Why doesn’t everyone use these languages? The drawback is that R and Python are both, at their core, programming languages.
When you start them up, you’re faced with a blinking cursor on a blank script. The learning curve can be very steep, which is why many people turn to other statistical analysis software packages.
Q: What is the easiest statistical analysis software to use?
A: Point and click statistical software tends to be the easiest to use: this accounts for the popularity of software like SPSS and GraphPad Prism.
While there is still a definite learning curve, you can fit statistical models using menu buttons and a drop-down menu. That can be easier to learn up front than using a script-based statistical analysis software.
The downside is that advanced features are not available, and data cleaning in point and click software is much harder than in script-based statistical analysis software.
Q: Is Microsoft Excel a statistical analysis software tool?
A: Yes, Excel offers a number of statistical analysis techniques, like ANOVA, linear regression, and chi-squared tests. Since Excel is already a popular office software both for data analysis and data visualization, it’s very tempting to use it for statistical analysis as well.
However, for anything but rudimentary statistics on small datasets, Excel can be slow and clunky. If you have large datasets, you can completely forget about using Excel: it can’t even open files with more than about a million rows.
Q: Can you do machine learning with statistical analysis software?
A: In the increasingly competitive statistical analysis software space, many companies are trying to distinguish themselves by incorporating machine learning capabilities into their software.
Even old-school stalwart software like SAS and SPSS are offering machine learning tools like clustering and support vector machines to stay competitive.
If your team is really serious about machine learning, you should opt for R or Python, since these are the two primary software packages used by machine learning teams in industry. If you just want to experiment with some basic techniques, other statistical analysis software will work as well.
Q: Can statistical analysis software do data visualization?
A: All statistical analysis software will be able to produce basic plots, but many are hampered by poor graphics and a limited set of visualization tools.
Many companies opt for one software package for statistical analysis, and another for data visualization. However, script-based statistical analysis software like R, Python, and MATLAB all offer support for high-quality, customizable visualizations.
The only drawback is that these statistical analysis software tools are harder to learn than graphical user interface tools.
Q: What kind of statistical analysis tools do you use for survey research?
A: Among companies that use survey research, a family of statistical techniques known as categorical data analysis is quite important.
Survey questions often come in the form of categories, such as “do you agree or disagree with the following statement…,” and this type of data requires special analysis.
For survey research, SPSS and SAS are probably the most popular, although for large datasets, R has many powerful features for categorical data as well.
Q: Can you use statistical analysis software for big data?
A: Yes, statistical analysis on big data gets complicated because the datasets are too large to fit into your computer’s active memory.
In general, script-based statistical analysis software like R, MATLAB, and Python are exceptionally well-suited for big data, while point and click software like SPSS or Microsoft Excel can struggle mightily or even fail completely on large datasets. Many advanced machine learning tools that are popular for big data are also only available in software like R, MATLAB, and Python.
If you know you are going to be doing statistics on big data, check to see what tools are available and what the computing capabilities of your software of choice are: as mentioned earlier, Excel is a particularly bad choice for big data, as it only supports about a million rows of data, while MATLAB can work on arbitrarily large datasets by working on them remotely, without loading the entire dataset into memory at once.
Q: What statistical analysis software should you use for multiple variable analysis?
A: Most statistical software will support multiple variables in your statistical analysis, though you should be careful to differentiate multiple variable statistical analysis from multivariate statistical analysis.
If you want to know how a consumer’s age, sex, and occupation are associated with how much money they spend at your grocery store, that’s multiple variable analysis, and can be done in any statistical analysis software.
If you want to simultaneously study how these factors are associated with spending on produce, meat, and frozen foods, that’s multivariate statistics. Not all statistical software supports this second case with as much ease as the first case.
Q: Is it hard to use statistical analysis software?
A: Among different types of business software, statistical analysis software probably has the most notorious reputation for being user unfriendly.
A big part of this reputation is probably because statistical analysis itself is difficult, math-intensive, and often very counter-intuitive. Another issue is that most statistical analysis software packages are written by statisticians and for statisticians.
Some software packages that use point and click graphical user interfaces are set up specifically to be easier for non-statisticians to use, but you still need to learn the specifics of the program, so you should prepare yourself for a solid period of learning when you start using a new statistical analysis software.
Recap
If you want to make robust conclusions from your company’s data, you need to use statistical analysis software. Without a formal statistical analysis, your inferences will be no better than a hunch.
With the right statistical analysis software, you can provide empirical and reproducible answers to key questions behind business decisions. Moreover, you can express your level of uncertainty in your answers, which is an advantage you can only get from the proper application of statistical techniques.
